Tip #26: Getting rid of ^M - mixing dos and unix

tip karma

Rating 299/121, Viewed by 13449

created:		March 5, 2001 17:00		complexity:		basic
author:		[email protected]		as of Vim:		5.7

If you work in a mixed environment you will often open files that have ^M's in them. An example would be this: ------------------------------------------------------------------ import java.util.Hashtable; ^M import java.util.Properties; ^Mimport java.io.IOException; import org.xml.sax.AttributeList; ^M import org.xml.sax.HandlerBase; ^Mimport org.xml.sax.SAXException; /**^M * XMLHandler: This class parses the elements contained^M * within a XML message and builds a Hashtable^M [snip] ------------------------------------------------------------------ Notice that some programs are not consistent in the way they insert the line breaks so you end up with some lines that have both a carrage return and a ^M and some lines that have a ^M and no carrage return (and so blend into one). There are two steps to clean this up. 1. replace all extraneous ^M: :%s/^M$//g BE SURE YOU MAKE the ^M USING "CTRL-V CTRL-M" NOT BY TYPING "CARROT M"! This expression will replace all the ^M's that have carriage returns after them with nothing. (The dollar ties the search to the end of a line) 2. replace all ^M's that need to have carriage returns: :%s/^M/ /g Once again: BE SURE YOU MAKE the ^M USING "CTRL-V CTRL-M" NOT BY TYPING "CARROT M"! This expression will replace all the ^M's that didn't have carriage returns after them with a carriage return. Voila! Clean file. Map this to something if you do it frequently. :help ffs - for more info on file formats thanks to jonathan merz, douglas potts, and benji fisher

<<color highlighting on telnet (esp w/ SecureCRT) | Convert hex to dec >>

Additional Notes

[email protected], October 22, 2001 8:51

It also works with :%s/\r//g

[email protected], January 29, 2002 9:01

If you want to put a map of this in your _vimrc file, then you have to double the ^V character so that it looks like this: map M :%s/^V^V^M$//g^M note: if you cut and paste this line, it won't work. You have to type the control characters as explained above. It has something to do with it being interpreted multiple times, I think. Roger

[email protected], March 10, 2002 17:41

:g/^M$/s///g - will get rid of every instance of a line ending with the carriage returns as well. You can map this to a key, but you need to use ^V to "quote" the carriage return and not by simply copying and pasting the above regular expression. Mark

[email protected], April 1, 2002 0:15

yet another way of doing this without the search is ':set fileformat=unix<cr>:w<cr>'... converts the crlfs to just lfs

[email protected], October 31, 2002 12:31

I think using this command is easier. :set ff=unix //to unix file :set ff=dos //to windows file

[email protected], January 22, 2003 19:00

Hi, yet another way to do this in a fine manner is made by 2 steps: first: %s/\r\r/\r/g second %s/\r//g It seems useless to do the first step on the first sight, but so it keeps your linebreaks as the document originally had - might be usefull. Arno

[email protected], February 26, 2003 15:31

[email protected] mentioned :set ff=unix //to unix file :set ff=dos //to windows file to change linebreaks. I experienced that this will only work on correct unix/dos files. If you have a mixture of linebreak types (if you see this ^M at then end of several lines), ":set ff" (fileformat) will not help you.

[email protected], May 28, 2003 9:06

This is more of a question than additional tips. How (if possible) can one remove the newline character from the LAST line?

Anonymous, June 19, 2003 15:36

You should be able to do: :$,$s/^M// But I have not tested that.

[email protected], June 25, 2003 6:46

When editing files on my Mac (OS X 10.2) then moving those to my Windows box, I get the crazy control M characters as well. I found out through trial and error the best way for me to get rid of them (and keep the original line breaks) is to use this command: :%s/\r/\r/g Basically says to get rid of the linefeeds you have, and put in linefeeds this system understands. Hope this helps someone out!

Anonymous, November 18, 2003 18:11

I am editing the same file under Linux, and Windows. I want just to hide the ^M, not actually erase them from the file. All the solutions that you suggested actually erase the ^M. It means that when I reopen the file under Windows, it's all messed up. If I don't change the ^M, it's painful to watch them in Vi 6.2. under Linux. Any solutions to this?

Anonymous, November 26, 2003 15:31

Note that, when using Vim on Windows, you will need to use CTRL-Q instead of CTRL-V to escape the special character on the ex command line. From the documentation (usr_24.txt): Note: On MS-Windows CTRL-V is used to paste text. Use CTRL-Q instead of CTRL-V. On Unix, on the other hand, CTRL-Q does not work on some terminals, because it has a special meaning.

Anonymous, February 3, 2004 12:30

To the poster who is editing files on multiple OSs: Try reading up on :help ffs. If your files truly are correct "DOS" (carriage return+newline) format when you open them in linux, VIM should be able to auto-detect the newline scheme and hide the actual control characters from you. Make sure the autodetection is enabled, or try forcing the file format from the command line when opening the file. If you still get ^M's, then the files you are opening have a mix of newline schemes and VIM is very inflexible when it comes to mixing newline schemes. This mixture often happens in log/data files and rolling files which sometimes never have a carriage return on the last line or silly things like that which break VIM's autodetection routine. Again try forcing the filetype from the command line. If this fails, your only option is to strip out the ^M's using the tips in this section.

Anonymous, February 3, 2004 12:32

[email protected], February 7, 2004 19:07

with: :%s/^M/\r/g works perfectly !!!

[email protected], June 18, 2004 18:24

if you wanted to do this to more then one file, open the files in vim with a command like vim *.txt start recording to the q register with the command qq remove the ^M's with :%s/^M$//g :%s/^M/ /g move to the next file :wnext stop recording q execute the q register ( to test it) @q if all works well run the command 999@q this will execute the q register on all the opened files, and will stop with an error when it gets to the last line. so basicaly the commansd go like this vim *.txt qq :%s/^M$//g :%s/^M/ /g :wnext q @q 999@q

Anonymous, June 18, 2004 18:25

oops, line=file

[email protected], June 21, 2004 10:33

Just save the following script as for example dosbreaks.sh ---------------------------------------------------------------------------------------- #!/bin/bash # To replace dos linebreaks for Unix compatibility echo "This script will replace the ^M line breaks from dos." echo -n "Enter filename without extension: " read file echo -n "Enter extension: " read ext sed 's/\r//' $file.$ext > $file2.$ext cp -f $file2.$ext $file.$ext rm -f $file2.$ext --------------------------------------------------------------------------------------- to run it, type ./dosbreaks.sh basically, you can just replace the ^M with sed sed 's/\r//' xxx.ext > xxx2.ext then copy xxx2.ext back to the original file xxx.ext and then remove the file xxx2.ext

[email protected], June 21, 2004 12:19

This script is the same as before, just minus one step. #!/bin/bash # To replace dos linebreaks for Unix compatibility echo "This script will replace the ^M line breaks from dos." echo -n "Enter filename: " read file sed 's/\r//' $file > 2$file cp -f 2$file $file rm -f 2$file

[email protected], July 28, 2004 11:00

There is this method that I generally employ. I use the command dos2unix file file e.g. if I have a file abc.txt with such ^M characters, simply write dos2unix abc.txt abc.txt at command prompt.




	If you have questions or remarks about this site, visit the vimonline development pages. Please use this site responsibly. Questions about Vim should go to [email protected] after searching the archive. Help Bram help Uganda.