Dismiss Notice
Join Physics Forums Today!
The friendliest, high quality science and math community on the planet! Everyone who loves science is here!

Merging columns from two text files

  1. May 25, 2016 #1


    User Avatar
    Gold Member

    Suppose I have two .txt files:
    Code (Python):

    X    Y    Z
    x1  y1   z1
    x2  y2   z2
    xN  yN  zN

    W   R
    w1   r1
    w2   r2
    wN   rN
    Is it possible to merge the collumns in a single txt file like:
    Code (Python):

    X    Y    Z   W   R
    x1  y1   z1   w1  r1
    x2  y2   z2   w2  r2
    xN  yN  zN   wN  rN
    Either with some c++ or python code, or with bash or vim.
  2. jcsd
  3. May 25, 2016 #2
    Yes, the first but maybe inefficient solution that comes to my mind is as follows,
    You read the first file and put the values into a 2D array(think of it as a matrix) and then you can read the second file and put the values into the corresponding index of the 2D array and then you can write a new file from this array.
  4. May 25, 2016 #3


    User Avatar
    Science Advisor
    Homework Helper
    2017 Award

    I hate excel, but for this kind of thing I would be inclined to use it.
  5. May 25, 2016 #4


    User Avatar
    Education Advisor

    A simple way to do this in python is to use pandas.

    Code (Text):

    >>> import pandas as pd

    >>> text1 = pd.read_csv("text1.txt", sep = "\t")

    >>> text1

        X   Y   Z

    0  x1  y1  z1

    1  x2  y2  z2
    >>> text2 = pd.read_csv("text2.txt", sep = "\t")

    >>> text2

        W   R

    0  w1  r1

    1  w2  r2
    >>> result = pd.concat([text1,text2],axis = 1)

    >>> result

        X   Y   Z   W   R

    0  x1  y1  z1  w1  r1

    1  x2  y2  z2  w2  r2

    result.to_csv("save_me.txt",sep = "\t")
    Also if you want to use bash then this works too:

    Code (Text):

    paste text1.txt text2.txt | awk '{print $1,$2,$3,$4,$5}'

    or if you want to write it out to another file.

    paste text1.txt text2.txt | awk '{print $1,$2,$3,$4,$5}' > result.txt
    Last edited: May 25, 2016
  6. May 25, 2016 #5


    Staff: Mentor

    Assuming that the two files are laid out as you show, with three columns of data in text1.txt and two columns of data in text2.txt, and both files have N lines of data, here is an algorithm:

    Code (Text):
        Read x, y, z from text1.txt
        Read w, r from text2.txt
        Write x, y, z, w, r to merged.txt
    While (text1.txt contains data AND text2.txt contains data)
  7. May 25, 2016 #6
    Did you say vim? As in one time shot with an editor? I don't know vim, but good editors like Notepad++ (on Windows), Nedit (on Linux) and others will allow you to select the two columns of the one file as a rectangular selection; then, you simply paste on the other file.
  8. May 26, 2016 #7


    User Avatar
    Gold Member

    I said vim because I know that vim allows a visual block selection (I guess it's the same as the rectangular selection you mentioned)...
    So I was guessing that if it's able to select blocks, it should also be able to paste them in some particular position (like an appending of collumn).
  9. May 26, 2016 #8

    jim mcnamara

    User Avatar

    Staff: Mentor

    bash means Linux. There are a lot of excellent text manipulation utilities in Linux. My point is do not write code when a command does it.

    Try paste. Example shell script - you can simply run the one line "paste .." command on the command line.
    Code (Text):

    paste test1.txt text2.txt > newfile.txt
    IMO writing code for this is counterproductive.
  10. May 27, 2016 #9
    Parallel processing library of C# I really love works like a charm. You should give it a try! :wink:

    private static void TextFileMerge(string fPath1, string fPath2, string fOutPath)
                try {
                    string[] fileContent1 = File.ReadAllLines(fPath1);
                    string[] fileContent2 = File.ReadAllLines(fPath2);
                    if (fileContent1.Length != fileContent2.Length)
                        throw new ArgumentException("Lengths of two files are different");
                    string[] fileMerge = new string[fileContent1.Length];
                    using (StreamWriter sw = new StreamWriter(fOutPath))
                        Parallel.For(0, fileContent1.Length, x =>
                            fileContent1[x] += " " + fileContent2[x];

                        foreach (string s in fileContent1)
                catch (Exception)
  11. May 29, 2016 #10


    User Avatar
    Gold Member

    I would import to Excel, then copy and paste. Might require only minutes.
  12. May 29, 2016 #11


    User Avatar
    Gold Member

    Is this something you need to automate or just an occasional need? If it is an occasional need find a text editor that does rectangular select. You can just copy and paste the data. For Windows I recommend Notepad++.

  13. May 30, 2016 #12


    User Avatar
    Gold Member

    1. needs excel and too much manual work (txt->excel->txt).
    2. excell is not able to accept columns. I think it would put everything into a single cell.

    yes... I want the program to do the calculations and stuff, the result is saved into text files in a form that can be immediately be compiled into a latex table.
    Because the code I am running has problems with closing files I get a system error of too many opened files. In order to overcome this problem I run the code several times with less file inputs. SO, the results are saved into different txt files which I want to merge to compile.
    Last edited: May 30, 2016
  14. May 30, 2016 #13


    Staff: Mentor

    When you are finished with a file, are you closing it? Since you started this thread asking about C++ and python, my answer is in that context. If you use either of those languages (and others besides those two), good housekeeping dictates that you close a file once you're done with it.
  15. May 30, 2016 #14


    User Avatar
    Gold Member

    No, there was generally an issue with how the code handled the open files. I know they solved the problem in the new version of their FW but my analysis is based on the old one [when I tried to update funny things happened].
  16. May 30, 2016 #15


    Staff: Mentor

    Which code are you talking about?
    What is FW?
Know someone interested in this topic? Share this thread via Reddit, Google+, Twitter, or Facebook

Have something to add?
Draft saved Draft deleted

Similar Discussions: Merging columns from two text files
  1. Input from text file (Replies: 13)