j"1LAnZuoQdcKCkpDBKQMCgziGMoPC4VQUckMI <div class="post"><div class="quoteheader"><a href="https://bitcointalk.org/index.php?topic=1931.msg24438#msg24438">Quote from: satoshi on November 25, 2010, 05:51:39 PM</a></div><div class="quote">Building blkindex.dat is what causes all the disk activity.<br/>[...]<br/>Maybe Berkeley DB has some tweaks we can make to enable or increase cache memory.<br/></div><br/>The following code in AddToBlockIndex(main.cpp) is horribly inefficient, and dramatically slows initial block download:<br/><br/><div class="codeheader">Code:</div><div class="code"> CTxDB txdb;<br/> txdb.WriteBlockIndex(CDiskBlockIndex(pindexNew));<br/><br/> // New best<br/> if (pindexNew->bnChainWork > bnBestChainWork)<br/> if (!SetBestChain(txdb, pindexNew))<br/> return false;<br/><br/> txdb.Close();<br/></div><br/>This makes it impossible to use a standard technique for loading large amounts of records into a database (db4 or SQL or otherwise): wrap multiple record insertions into a single database transaction. Ideally, bitcoin would only issue a TxnCommit() for each 1000 blocks or so, during initial block download. If a crash occurs, the database remains in a consistent state.<br/><br/>Furthermore, <b>database open + close for each new block</b> is incredibly expensive. For each database-open and database-close operation, db4<br/><ul style="margin-top: 0; margin-bottom: 0;"><li>diagnose health of database, to determine if recovery is needed. this test may require data copying.</li><li>re-init memory pools</li><li>read database file metadata</li><li>acquire file locks</li><li>read and initialize b-tree or hash-specific metadata. build hash table / b-tree roots.</li><li>forces a sync, even if transactions called with DB_TXN_NOSYNC</li><li>fsync memory pool</li></ul><br/>And, additionally, bitcoin forces a database checkpoint, pushing all transactions from log into main database.<br/><br/>That's right, that long list of operations is executed per-database (DB), not per-environment (DB_ENV), for a database close+open cycle. To bitcoin, that means we do this for <i>every new block</i>. Incredibly inefficient, and not how db4 was designed to be used.<br/><br/>Recommendations:<br/><br/>1) bitcoin should be opening databases, not just environment, at program startup, and closing database at program shutdown. db4 is designed to handle crashes, if proper transactional use is maintained -- and bitcoin already uses db4 transactions properly.<br/><br/>2) For the initial block download, txn commit should occur once every N records, not every record. I suggest N=1000.<br/><br/><br/><br/>EDIT: Updated a couple minor details, and corrected some typos.</div> text/html
https://whatsonchain.com/tx/1e01b77b3fb97cc3ddedcf2f739eddaf9f0ae0d4d8b4e47e12670e27e24573a5