Import Wikipedia Dump into MySQL 5.0 w/o Heat Issues

  • Thread starter sid_galt
  • Start date
  • Tags
    Wikipedia
In summary, the problem is that the temperature of my hard disk is reaching 50C (120F) after just 30000 pages.
  • #1
sid_galt
502
1
I am trying to import the a wikipedia 5.6 GB XML dump into MySQL 5.0 using mwdumper. The problem is that the temperature of my hard disk is reaching 50C (120F) after just 30000 pages.

Is there any way to prevent this or to import the wikipedia dump into MySQL in parts?
Speed is not an issue so if there is a solution which can solve my problem but imports at a low speed, I wouldn't mind it.

PS - I am getting speed 35 pages/sec in mwdumper and 8 pages/sec in import.php.
 
Last edited:
Computer science news on Phys.org
  • #2
That sounds very interesting. What kind of data are you importing?
 
  • #3
Buy a fan for the HD.
 
  • #4
Or put your computer in your refrigerator temporarily, if it's at all feasible. Server rooms are usually refrigerated for this reason.
 
  • #5
-Job- said:
That sounds very interesting. What kind of data are you importing?

The wikipedia (en.wikipedia.org) encyclopedia, nothing special.

-Job- said:
Or put your computer in your refrigerator temporarily, if it's at all feasible. Server rooms are usually refrigerated for this reason.

There is no way to route the comp cables out of the refrigerator without keeping it open, so not possible.Do you mean then that there is no way to import the data in parts? Maybe by modifying the script to filter pages according to the first alphabet of the title page and then execute the script for each of the 26 alphabets?
 
  • #6
Or modify the script so you can pause it, or even pause the process. I don't know much about the import method so i can't be much help.
 
  • #7
There's a way to do it so that it doesn't have to be done all at once, but exactly how depends on the structure of the XML file. It's not a terribly complicated process. You read from the XML file and insert into the database server. Perhaps one way to do this is to manually split the XML file into smaller pieces, then you can still use whatever script you're using rather than write one, but again that depends on the structure of the XML.
 
  • #8
Box fan...
 

1. How do I import a Wikipedia dump into MySQL 5.0 without causing heat issues?

To import a Wikipedia dump into MySQL 5.0 without heat issues, you will need to follow these steps:

  • First, make sure you have enough disk space and memory available to handle the large dump file.
  • Next, create a new database in MySQL 5.0 to store the Wikipedia data.
  • Use a tool such as mysqlimport or LOAD DATA INFILE to import the dump file into the database.
  • Set the innodb_buffer_pool_size variable in MySQL to a higher value to prevent heat issues.
  • Finally, monitor the import process and adjust the innodb_buffer_pool_size value as needed to prevent overheating.

2. What is the best way to handle a large Wikipedia dump when importing into MySQL 5.0?

The best way to handle a large Wikipedia dump when importing into MySQL 5.0 is to use a tool such as mysqlimport or LOAD DATA INFILE. These tools are specifically designed for importing large data sets into MySQL and can handle the size and complexity of a Wikipedia dump more efficiently than manual methods.

3. How can I prevent heat issues when importing a Wikipedia dump into MySQL 5.0?

To prevent heat issues when importing a Wikipedia dump into MySQL 5.0, you will need to adjust the innodb_buffer_pool_size variable. This controls the amount of memory allocated to the InnoDB buffer pool, which is used to cache data and indexes. By increasing this value, you can prevent overheating and improve the performance of the import process.

4. What should I do if I encounter heat issues during the Wikipedia dump import process?

If you encounter heat issues during the Wikipedia dump import process, you should first check the status of the innodb_buffer_pool_size variable. If it is already set to a high value, you may need to allocate more memory to your server or adjust other MySQL settings to improve performance. You can also try using a tool such as mysqlimport with the --compress option to compress the data during the import process and reduce the strain on your server.

5. Is it possible to import a Wikipedia dump into MySQL 5.0 without experiencing heat issues?

Yes, it is possible to import a Wikipedia dump into MySQL 5.0 without experiencing heat issues. However, it may require adjusting the innodb_buffer_pool_size variable and closely monitoring the import process to prevent overheating. Using a tool like mysqlimport can also help improve the efficiency of the import and reduce the risk of heat issues.

Similar threads

Requesting suggestions for languages, libraries, and architectures for parallel (and sometimes non parallel) numerical and scientific computations
    • Programming and Computer Science
Replies
8
Views
2K
STOP: 0x0000008E (0xC000005, 0XBF04DF7B, 0xED924B18 0X000000000)
    • Computing and Technology
Replies
23
Views
5K
Mathematica This Week's Finds in Mathematical Physics (Week 261)
    • MATLAB, Maple, Mathematica, LaTeX
Replies
5
Views
2K
Mathematica This Week's Finds in Mathematical Physics (Week 259)
    • MATLAB, Maple, Mathematica, LaTeX
Replies
1
Views
2K

Hot Threads

Recent Insights

Back
Top