Need Help Turning One File into Thousands of Files (in Python)

In summary, the conversation discusses the process of writing a Python script to take in a file as an input, transform it into a list, and output every 567 lines to new separate files. The speaker acknowledges their lack of programming experience and asks for guidance on the necessary steps and functions to accomplish this task. A solution involving a while loop and exception handling is suggested by the responder, along with the use of numerical indices in file names. Debugging and error handling are also mentioned as important considerations.
  • #1
ADCooper
20
1

Homework Statement



I'm not sure if this is quite where this belongs since it's not homework really, but I just need help programming and I figured this would be the place to ask. It's related to undergraduate research I'm doing so if that's not kosher just say so!

I want to write a Python script that takes in a certain file as an input, and then, after transforming the the file as a list where each line is an index, then outputting every 567 lines (so something like [0:566], [567:1133], etc] to new separate files. There are well over a million lines if I remember correctly so doing this without a script would be extremely inefficient (unless I've overlooked some super simple way to do it in the bash shell). I only need to do this for one file, so technically I don't need to find an extremely general solution for this, just one that works with this file.

The problem is that I have very little programming experience, so a little guidance would be super helpful (even just basic programming advice is certainly welcome). I don't want someone to do it for me because I need to be able to do this myself, I just need to know if my ideas will possibly work and what general functions I should read up on to get this accomplished if I'm on the complete wrong track.

If this is the wrong place for this I apologize!2. List of Things that Must Be Done

1. Accept name of text file as an argument when executing file.

2. Turn Text File type into a python list. (I'm not sure if this is even necessary. Are text files already considered as a "list" type when they are opened in a python environment? Are they already indexed based on line?)

3. Output every 567 lines to separate text files. (Could technically do this some way by using a keyword that will start every separate file, but I'd assume that would require more work)


The Attempt at a Solution



1. This one I basically understand. Simply have to import the necessary packages and set the first argument equal to the file being split

2. I'm not really sure yet if this is even necessary because I don't know if the file, after being called on the command line, is already indexed by line. Any clues on this?

3. Perhaps a for loop (something like for line [i,j] in file) with i = 0 and j = 566. If I can find a way within the loop to call i and j the indices which I would then use some way to output to a file of some form FILE#, and with every iteration of the loop increase the # next to the file name by 1 and increase i and j by 567. Am I on the right track? Or should I abandon ship ont his way and start from scratch?
 
Technology news on Phys.org
  • #2
You'll want a loop, but you may find a while loop easier to work with and use exception handling:

Code:
inport sys

fIn = open('mybigfile.txt')
while True:
     try:
          "Inner loops here will open output file (fOut = open('filename1.txt')   ), 
           read lines and write to file, 
           close file and open next file as if it would go on forever."
     except IOError:
           "when end of file is reached read function will throw an error
            which this code catches.  
            Here we close last file and break out of loop."
             fOut.close()
             break

print "All Done!"

Note to use numerical index in file names you can use the str function and string operations:
e.g. open('filename'+str(k)+'.txt') will when k=5 give you: open('filename5.txt')

EDIT: You may want to include some debugging progress code, say print a message saying "opening output file"+filename each time you open a new file. You might want to be more precise with the exception handling, making sure it is an EOF and not some other file error, at least say, print out the error type. Check online python documentation for this.
 
  • #3
THanks I really appreciate the help!
 

Related to Need Help Turning One File into Thousands of Files (in Python)

1. How can I turn one file into thousands of files using Python?

There are a few different ways to accomplish this task in Python. One approach is to use the os module to create new files and the shutil module to copy the contents of the original file into each new file. Another approach is to use the glob module to generate a list of filenames and then use a loop to create and populate each new file.

2. Can this process be automated?

Yes, this process can be automated by writing a Python script that can be run on a schedule or as needed. The script can take in user inputs such as the name of the original file, the desired number of new files, and any desired naming conventions for the new files.

3. Are there any limitations to the number of files that can be created?

The limitations will depend on the capabilities of your computer and the size of the original file. If you are working with a large file, it is important to consider the available storage space on your computer before attempting to create thousands of new files.

4. Can I specify the contents of each new file?

Yes, you can specify the contents of each new file by using Python's file handling functions. You can read in the contents of the original file, manipulate them as needed, and then write them to each new file.

5. Are there any alternatives to using Python for this task?

Yes, there are other programming languages and tools that can be used to accomplish this task. For example, you could use a command-line tool like split in UNIX or a scripting language like Bash to split a file into multiple files. However, if you are already familiar with Python, it is a great option for automating this task.

Similar threads

  • Programming and Computer Science
2
Replies
35
Views
2K
  • Programming and Computer Science
Replies
2
Views
758
Replies
6
Views
1K
  • Programming and Computer Science
Replies
12
Views
8K
  • Programming and Computer Science
Replies
5
Views
1K
  • Programming and Computer Science
Replies
16
Views
3K
  • Programming and Computer Science
Replies
10
Views
2K
  • Programming and Computer Science
Replies
29
Views
2K
  • Programming and Computer Science
Replies
8
Views
995
  • Programming and Computer Science
Replies
1
Views
686
Back
Top