QNX & SQLite (On NAND flash drive)

In summary: This might help with performance.But basically setting it to 3 SQLite will keep all temp files in memory as opposed to writing the temp files out to the disk.
  • #1
403
120
Wondering if anyone knows anything about this.

It was mentioned to me in passing. That on QNX, with an ETFS file system when you append to a file, that it completely rewrites that entire file to another location on the Drive. This presenting an issue with extra NAND flash writes ultimately causing corruption. (if anyone has more info on this that would be great as well)

So my question is, when you do an insert into a SQLite DB, and SQLite writes that insert to the disk, is that going to do an append and a complete rewrite?
We want to ensure that we arent' going to corrupt drives 6 months down the road here :)

Thanks for your help!
 
Technology news on Phys.org
  • #2
I don't see why it would cause any corruption as long as the file allocation information is properly handled (which it certainly should be).
 
  • #3
phinds said:
I don't see why it would cause any corruption as long as the file allocation information is properly handled (which it certainly should be).
Because of the finite number of write cycles to a flash drive before it wears out. On the order of 104 - 105 writes to a location.
 
  • #4
Tom.G said:
Because of the finite number of write cycles to a flash drive before it wears out. On the order of 104 - 105 writes to a location.
Well, sure but that's a function of just WRITING (which obviously IS done more if you re-write entire files) not something that is caused directly by rewriting rather than appending. That is, writing lots of new files and then "erasing" them does the same thing. I put "erasing" in quotes because files are not actually erased, they just get their disk space taken out of the file allocation pointers and the space is used in rotation with other unused space so as to minimize writing to any given cell.
 
  • #5
Why this was raised was the thought that appending to these files would cause the OS to copy the entire file to a new physical location and then add the little extra bit it.

Only reason I could think of why the OS would do this is to keep the drive non-fragmented to maintain performance.
 
  • #6
cpscdave said:
Why this was raised was the thought that appending to these files would cause the OS to copy the entire file to a new physical location and then add the little extra bit it.

Only reason I could think of why the OS would do this is to keep the drive non-fragmented to maintain performance.
With NAND flash device architecture, individual bytes are not addressable. Writes are for at least a page/block at a time. The number of writes is limited, so in order to avoid uneven wear, writes are preferentially targeted to areas in inverse order of of how many times the area has previously been written. As you surmised, in order to keep access speeds optimal, files are preferentially written to contiguous areas. To maintain contiguity, systems will, at least for a file composed of a small number of pages/blocks, write the entire file for an update, even if the update was to content within only one block of a multi-block file.
 
  • Like
Likes Tom.G
  • #7
I looked into this a bit more and it looks as if you can compile SQLite to work with NAND drives
by setting -DSQLITE_TEMP_STORE=3

More info can be found here: https://www.sqlite.org/compile.html

But basically setting it to 3 SQLite will keep all temp files in memory as opposed to writing the temp files out to the disk.
 
  • Like
Likes Tom.G

1. What is QNX?

QNX is a real-time operating system that is known for its reliability, scalability, and security. It is widely used in embedded systems, including in industries such as automotive, aerospace, and healthcare.

2. What is SQLite?

SQLite is a popular open-source relational database management system that is known for its lightweight design and ability to operate without a server, making it a popular choice for embedded systems. It is widely used in mobile apps and other small-scale applications.

3. How are QNX and SQLite used on NAND flash drives?

QNX and SQLite can be used together on NAND flash drives to create a robust and efficient data storage solution for embedded systems. QNX provides the real-time operating system functionality, while SQLite manages the data storage and retrieval on the NAND flash drive.

4. What are the benefits of using QNX and SQLite on NAND flash drives?

Using QNX and SQLite on NAND flash drives offers several benefits, including fast and efficient data storage and retrieval, low power consumption, and high reliability. Additionally, the combination of QNX and SQLite allows for easy integration and customization for specific embedded system needs.

5. Are there any limitations to using QNX and SQLite on NAND flash drives?

One limitation of using QNX and SQLite on NAND flash drives is the limited storage capacity of the NAND flash drive itself. Additionally, the use of a real-time operating system like QNX may require some learning and adaptation for developers who are used to working with traditional operating systems.

Suggested for: QNX & SQLite (On NAND flash drive)

Back
Top