Jekyll2023-06-05T14:37:40-05:00https://blog.ipfs-search.com/feed.xmlipfs-search.com blogProject blog for ipfs-search.comipfs-search.comBump in the road2023-06-02T00:00:00-05:002023-06-02T00:00:00-05:00https://blog.ipfs-search.com/bump-in-the-road<p>Alas. We are shutting down ipfs-search, with a full-stop on june the 7th. For now. The upkeep is too much to carry and we have not found the necessary support yet. We don’t know when we will go live again, there is no certainty at this point. Only reflection with a sense of pride of what we built and achieved over the years, and hope for a relaunch.</p>
<h3 id="our-journey">Our journey</h3>
<p><a href="http://Ipfs-search.com">ipfs-search.com</a> started in 2016 as an idea of Mathijs to use IPFS to index his ebooks collection using IPFS. This quickly grew with a crawler that “sniffed” the updates of IPFS, collecting what was being shared on the entire network by everybody. Add metadata-extraction, a search API and a simple frontend, ipfs-search was born.</p>
<p><img src="/assets/images/2023-06-02-bump-in-the-road/2023-06-02-First-frontend.png" alt="Untitled" /></p>
<p>Later, Aad from <a href="http://RedPencil.io">RedPencil.io</a> joined to help with frontend improvements, hosting, and fundraising, and over the years, <a href="http://Ipfs-search.com">ipfs-search.com</a> grew to a usable search engine for IPFS, that took a stance against collecting users’ personal data or promoting search results based on advertisements. In fact, there are no advertisements and no user-targeted biases in the search results. We store nothing about our users!</p>
<p>At the end, the index grew with a whopping 1 million CIDs per day, and with help of an <a href="https://nlnet.nl/NGI0/">NLNet/NGI0 grant</a> we created the third iteration of the frontend, completely equipped with search filters, well-designed layout, mobile-friendly, fileviewers for almost anything including e-books, images, videos, and even a music player with playlists.</p>
<p>We received a <a href="https://github.com/filecoin-project/devgrants/blob/master/open-grant-proposals/ipfs-search-scale-out.md">Filecoin Dev Grant</a> to make the search engine scaleable for a lot of user traffic, for which we had to overcome several <a href="https://blog.ipfs-search.com/challenge-accepted/">bumps in the road</a> and finally succeeded. The search engine is now ready for next stages, and gives a great way to access content on ipfs with an unparalleled index of data collected over 7 years.</p>
<p><img src="/assets/images/2023-06-02-bump-in-the-road/2023-06-02-Newest-frontend.png" alt="Untitled" /></p>
<p>There are some problems too; while we had creeped in a lot of awesome technical features, the search is not biased and therefore does not really target any user segments. We had no marketing department to gather more users and unmoderated access to the millions of random files shared on IPFS do not generate viral adoption of the search engine. Without advertisement or user targeting as a business model there was no immediate way to monetize on the frontend as a product at all, let alone enough to work on usability features. We did not want to sell out on our principles and more funding appeared a lot harder to find than expected. The cryptowinter and the global economic turmoil did not help in that regard. In the meantime, the costs in servers and manhours are piling up.</p>
<h3 id="hope-for-the-future">Hope for the future</h3>
<p>So, as we are running into the red, we have decided to shut down, albeit with great regret, until we find a way to make this sustainable. There are a lot of people expressing their support and even helping to fund us through <a href="https://opencollective.com/ipfs-search">OpenCollective</a>, and we truly hate to feel like we are letting them down. Fortunately, there are some beacons of hope on the horizon!</p>
<p>First of all, we have found a new partner to take care of hosting ipfs-search.com, at <a href="https://www.notion.so/a6ef0ea4ea404079a2e4e2d051d95e6d?pvs=21">DCent</a>. They have offered generously to grant their hardware to us while we find new support, and we hope we can migrate there soon. Besides this, we are exploring several options of funding, for which we may have to develop novel ways to apply the technology or invent new features that make it attractive for future users.</p>
<h3 id="epilogue">Epilogue</h3>
<p>We hit a bump in the road, and have to shut down. But we hope it does not affect our users and supporters for too long, and we are making good plans to restart. We are looking forward to the next phase of this project and are proud of the efforts that have brought us so far, so keep an eye on our feeds. And if you have any idea or questions on how to help, contact us at info@ipfs-search.org</p>Frido EmansWe are shutting down ipfs-search.com. For good? We hope not. A retrospective and a look forward.The Crossroads: ipfs-search.com’s Fight for Survival2023-05-23T00:00:00-05:002023-05-23T00:00:00-05:00https://blog.ipfs-search.com/crossroads<h1 id="introduction">Introduction</h1>
<p>Since 2016, we at <a href="http://ipfs-search.com/">ipfs-search.com</a> have been committed to building a neutral, privacy-friendly, open-source search for the Web3 community. However, today we find ourselves at a critical crossroads. Financial challenges are threatening our ability to continue to operate, risking an imminent shutdown of our public services. We wanted to take this moment to reflect on our journey, explain our current situation, and outline our next steps.</p>
<figure>
<img alt="Crossroads" src="/assets/images/2023-05-23-crossroads/crossroads.jpg" />
<figcaption>Standing at the crossroads, ready to forge a path forward. CC BY-SA 2.0 <a href="https://www.flickr.com/photos/laenulfean/5943132296">Carsten Tolkmit</a></figcaption>
</figure>
<h1 id="our-journey-and-impact">Our Journey and Impact</h1>
<p>Our story began in 2016, as a hobby project with a vision for a more decentralized and democratized internet. We saw the potential of the Interplanetary File System (IPFS) to disrupt traditional web paradigms and provide a genuinely open, resilient, and privacy-focused infrastructure for information. Recognizing that IPFS’s potential would be unattainable without an effective search and discovery tool, we embarked on creating the first search engine specifically designed for IPFS, giving birth to ipfs-search.com.</p>
<p>As pioneers in this space, we committed to creating a search engine rooted in principles of privacy, neutrality, and transparency – ideals often overlooked by traditional search engines. Our aim has always been to make information accessible to all, unbiased, and free from the control of any single entity.</p>
<p>Throughout the years, we’ve had the pleasure of receiving support from various sources. We received backing from the EU commission’s <a href="https://nlnet.nl/discovery/">NGI0 Search and Discovery fund</a> through <a href="https://nlnet.nl/">NLNet Foundation</a>, an organization dedicated to promoting a networked world, unrestricted by commercial or political monopolies. We also had to face difficult times, including a previous shutdown, after which Aad from <a href="https://redpencil.io/">redpencil.io</a> graciously stepped up and offered to support our hosting temporarily.</p>
<p>This support enabled us to evolve from a single server to a proper cluster, enhancing our service. However, as our infrastructure grew, so did the associated costs. Operating costs have become prohibitive for our small initiative, despite our significant personal investments and dedication to this cause. The financial commitment required for maintaining our services has been coupled with our personal financial needs, exacerbated by the onset of the crypto winter and a wider economic downturn.</p>
<p>Nevertheless, our journey has been filled with remarkable achievements. From <a href="https://blog.ipfs-search.com/challenge-accepted/">handling 1000 hits/s on our API endpoints</a> in 2022, showcasing our readiness to scale with IPFS and handle large-scale integration as the first search and discovery platform within the IPFS ecosystem, to contributing to the broader Web3 movement, we’ve left an indelible mark on the digital landscape.</p>
<p>We’ve proudly fostered a more open and inclusive digital world through our open-source commitment, enabling the community to freely use, study, share and improve our work. But as we now stand at a critical crossroads, we reflect with pride on our accomplishments and look forward with resolve to surmount the challenges that lay before us. Our mission of providing a truly neutral, open-source search for Web3 remains unshakeable.</p>
<h1 id="the-current-situation">The Current Situation</h1>
<p>Without additional funding, we will need to shut down our public APIs in the coming weeks. Site search is already suspended, but API access remains—for now. This is a consequence of the balancing act between our commitment to the mission and the stark reality of operational costs and personal sustenance needs.</p>
<figure>
<img alt="Screenshot of our frontend being shutdown, with a banner instead of search." src="/assets/images/2023-05-23-crossroads/screenshot.png" />
<figcaption>Site search is already suspended, but API access remains—for now.</figcaption>
</figure>
<h1 id="the-road-ahead">The Road Ahead</h1>
<p>However, we view this not as the end of our journey, but as a challenging bend in the road. Our vision of a truly democratized internet, where information is accessible to all and uninfluenced by political or commercial interests, remains strong. We’re actively looking for solutions, seeking new funding sources, and exploring every possible avenue to continue our mission.</p>
<p>We need your help to navigate this. If you believe in what we do, please <a href="https://twitter.com/intent/tweet?text=%F0%9F%9A%A8URGENT%3A%20ipfs-search.com%2C%20trusted%20%23Web3%20search%20since%202016%2C%20is%20down%20due%20to%20financial%20challenges.%20Help%20safeguard%20the%20future%20of%20open%2C%20unbiased%20search%20for%20%23IPFS.%20Spread%20the%20word%20%26%20show%20your%20support%20at%20https%3A%2F%2Fopencollective.com%2Fipfs-search%20%23SaveIPFSSearch%20">share</a> our situation within your community and consider supporting us directly through <a href="https://opencollective.com/ipfs-search">OpenCollective</a>.</p>
<figure>
<img alt="Misty view of curvy mountain road." src="/assets/images/2023-05-23-crossroads/misty-future.jpg" />
<figcaption>We view this as a challenging bend in the road.</figcaption>
</figure>
<h1 id="stay-in-touch">Stay in Touch</h1>
<p>We are committed to transparency and will continue to share updates here on the blog and on <a href="https://mastodon.social/@ipfssearch">Mastodon</a> and <a href="https://twitter.com/SearchIpfs">Twitter</a>. We’re always open to your questions and suggestions, so feel free to reach out to us at <a href="mailto:info@ipfs-search.com">info@ipfs-search.com</a>.</p>
<p>Thank you for your understanding and unwavering support.</p>
<p>Sincerely,</p>
<p>The <a href="http://ipfs-search.com/">ipfs-search.com</a> Team<br />
Mathijs de Bruin<br />
Frido Emans<br />
Aad Versteden</p>Mathijs de BruinIn this critical update, we explain the urgent financial challenges facing ipfs-search.com, our response to these issues, and the steps we're taking to continue our mission. We detail our journey since 2016 and the crucial role we play in the Web3 and IPFS ecosystem.Searching Web 3 at Web Scale2023-04-18T00:00:00-05:002023-04-18T00:00:00-05:00https://blog.ipfs-search.com/searching-at-scale<h1 id="introduction">Introduction</h1>
<p>In 2021 we set ourselves the ambition of being able to handle 1000 hits/s on our API endpoints. To demonstrate that we are ready to scale with IPFS and that we can handle large-scale integration as the first and only search and discovery platform within the IPFS ecosystem.</p>
<figure>
<img alt="Source: https://messari.io/report/state-of-filecoin-q4-2022" src="https://cdn.sanity.io/images/2bt0j8lu/production/e992bf46793e79d3a5bad6ade311cba1752b027d-1920x1080.png?w=900&fit=max&auto=format&dpr=3" />
<figcaption>Source: <a href="https://messari.io/report/state-of-filecoin-q4-2022">https://messari.io/report/state-of-filecoin-q4-2022</a></figcaption>
</figure>
<p>This is the second post in a 2-post miniseries where we explain the challenges we faced scaling up, and how they were eventually overcome. In <a href="https://blog.ipfs-search.com/challenge-accepted/">the previous post</a> we dealt mainly with low response times as we scaled our cluster to 33 nodes. We will now describe how we built a realistic benchmark. It details the problems which we faced scaling up to 73 nodes and how they were overcome by completely restructuring our indexes.</p>
<p>Finally, we can say that our platform can handle well over 1300 hits/s with <150ms for 95% of requests, equivalent to serving 3100 unique users.</p>
<figure>
<img src="/assets/images/2023-04-18-searching-at-scale/Untitled.png" />
</figure>
<figure>
<img src="/assets/images/2023-04-18-searching-at-scale/Untitled%201.png" />
</figure>
<h1 id="building-a-real-world-benchmark">Building a real-world benchmark</h1>
<h2 id="caching-in-opensearch">Caching in OpenSearch</h2>
<p>As with any real-world web-application, our search engine and it’s backend OpenSearch, heavily rely on caching. Particularly, the <a href="https://lucene.apache.org/core/">Lucene</a> indexes on which OpenSearch is built uses <a href="https://dzone.com/articles/use-lucene%E2%80%99s-mmapdirectory">memory-mapped files</a>, transparently allowing the OS kernel to keep all or parts of the search index in RAM. This means that Lucene can read files as if they are already in memory. It also means that there is <em>simply no way to switch caching off</em>. On top of this, OpenSearch has caches for requests, for shard data and for sorting and aggregation (field data), which use heap memory. Hence the recommendation to allocate no more than about half of RAM to OpenSearch/Elasticsearch’s heap, the remainder being used by the OS’s VFS (<a href="https://en.wikipedia.org/wiki/Virtual_file_system">virtual file system</a>) to cache memory mapped files.</p>
<p>This is particularly relevant designing a benchmark for a large database or a search engine. If we were simply to repeat a single request, or a small number of requests, we would not really be benchmarking our search engine — we would be testing the performance of it’s caches instead!</p>
<h2 id="creating-benchmarks-from-real-world-traffic">Creating benchmarks from real-world traffic</h2>
<p>In order to circumvent this problem, we decided to use actual API requests to model ‘virtual users’. While <a href="https://github.com/ipfs-search/ipfs-search-deployment/blob/main/roles/vendor/nginx/templates/nginx.conf.j2#L32">we do not store any identifiable information</a> on our users (yay, no pesky GDPR banners!), we do in fact log all requests. So we <a href="https://github.com/ipfs-search/ipfs-search-benchmark/blob/main/logtobatches.js">wrote a script</a> and processed some 6 months of log data into visits; browsing experiences of virtual users, based on actual user journeys through our API and frontend.</p>
<p>The result is a 240 MB JSON blob and a short JavaScript file to be used with Grafana’s load tester <a href="https://k6.io/">k6</a>. You can check out our <a href="https://github.com/ipfs-search/ipfs-search-benchmark">repo</a> to see exactly what we’ve done!</p>
<h3 id="choosing-k6">Choosing k6</h3>
<p>We just love everything in Grafana’s stack, particularly how <a href="https://grafana.com/blog/2021/04/20/grafana-loki-tempo-relicensing-to-agplv3/">they’re AGPL</a>, like us! But we chose k6 because it’s extremly efficient in handling a large amount of parallel sockets, using Golang’s goroutines for fully non-blocking parallel performance while using a JS VM (<a href="https://github.com/dop251/goja">Goja</a>) for implementing/scripting the actual tests. This ensures that the machine doing the tests is almost never the bottleneck and hence’ (at this scale) we don’t have to worry about coordinating load tests from multiple machines.</p>
<h3 id="early-results">Early results</h3>
<p>With the tests we created, we can simply select the amount of Virtual Users (VU), or <a href="https://github.com/ipfs-search/ipfs-search-benchmark/blob/main/k6loadtest.js#L35">specify ramping in stages</a> to perform tests yielding results like this:</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="err">running</span><span class="w"> </span><span class="err">(</span><span class="mi">5</span><span class="err">m</span><span class="mf">30.0</span><span class="err">s),</span><span class="w"> </span><span class="mi">0000</span><span class="err">/</span><span class="mi">2000</span><span class="w"> </span><span class="err">VUs,</span><span class="w"> </span><span class="mi">5129</span><span class="w"> </span><span class="err">complete</span><span class="w"> </span><span class="err">and</span><span class="w"> </span><span class="mi">1604</span><span class="w"> </span><span class="err">interrupted</span><span class="w"> </span><span class="err">iterations</span><span class="w">
</span><span class="err">default</span><span class="w"> </span><span class="err">✓</span><span class="w"> </span><span class="p">[</span><span class="err">======================================</span><span class="p">]</span><span class="w"> </span><span class="mi">2000</span><span class="w"> </span><span class="err">VUs</span><span class="w"> </span><span class="mi">5</span><span class="err">m</span><span class="mi">0</span><span class="err">s</span><span class="w">
</span><span class="err">✗</span><span class="w"> </span><span class="err">is</span><span class="w"> </span><span class="err">status</span><span class="w"> </span><span class="mi">200</span><span class="w">
</span><span class="err">↳</span><span class="w"> </span><span class="mi">79</span><span class="err">%</span><span class="w"> </span><span class="err">—</span><span class="w"> </span><span class="err">✓</span><span class="w"> </span><span class="mi">173720</span><span class="w"> </span><span class="err">/</span><span class="w"> </span><span class="err">✗</span><span class="w"> </span><span class="mi">44841</span><span class="w">
</span><span class="err">✗</span><span class="w"> </span><span class="err">checks.........................:</span><span class="w"> </span><span class="mf">79.48</span><span class="err">%</span><span class="w"> </span><span class="err">✓</span><span class="w"> </span><span class="mi">173720</span><span class="w"> </span><span class="err">✗</span><span class="w"> </span><span class="mi">44841</span><span class="w">
</span><span class="err">data_received..................:</span><span class="w"> </span><span class="mi">516</span><span class="w"> </span><span class="err">MB</span><span class="w"> </span><span class="mf">1.6</span><span class="w"> </span><span class="err">MB/s</span><span class="w">
</span><span class="err">data_sent......................:</span><span class="w"> </span><span class="mi">35</span><span class="w"> </span><span class="err">MB</span><span class="w"> </span><span class="mi">107</span><span class="w"> </span><span class="err">kB/s</span><span class="w">
</span><span class="err">http_req_blocked...............:</span><span class="w"> </span><span class="err">avg=</span><span class="mf">47.9</span><span class="err">ms</span><span class="w"> </span><span class="err">min=</span><span class="mi">0</span><span class="err">s</span><span class="w"> </span><span class="err">med=</span><span class="mi">250</span><span class="err">ns</span><span class="w"> </span><span class="err">max=</span><span class="mf">21.83</span><span class="err">s</span><span class="w"> </span><span class="err">p(</span><span class="mi">90</span><span class="err">)=</span><span class="mi">400</span><span class="err">ns</span><span class="w"> </span><span class="err">p(</span><span class="mi">95</span><span class="err">)=</span><span class="mi">491</span><span class="err">ns</span><span class="w">
</span><span class="err">http_req_connecting............:</span><span class="w"> </span><span class="err">avg=</span><span class="mf">34.1</span><span class="err">ms</span><span class="w"> </span><span class="err">min=</span><span class="mi">0</span><span class="err">s</span><span class="w"> </span><span class="err">med=</span><span class="mi">0</span><span class="err">s</span><span class="w"> </span><span class="err">max=</span><span class="mf">15.54</span><span class="err">s</span><span class="w"> </span><span class="err">p(</span><span class="mi">90</span><span class="err">)=</span><span class="mi">0</span><span class="err">s</span><span class="w"> </span><span class="err">p(</span><span class="mi">95</span><span class="err">)=</span><span class="mi">0</span><span class="err">s</span><span class="w">
</span><span class="err">✓</span><span class="w"> </span><span class="err">http_req_duration..............:</span><span class="w"> </span><span class="err">avg=</span><span class="mf">1.25</span><span class="err">s</span><span class="w"> </span><span class="err">min=</span><span class="mi">0</span><span class="err">s</span><span class="w"> </span><span class="err">med=</span><span class="mf">1.54</span><span class="err">ms</span><span class="w"> </span><span class="err">max=</span><span class="mi">1</span><span class="err">m</span><span class="mi">0</span><span class="err">s</span><span class="w"> </span><span class="err">p(</span><span class="mi">90</span><span class="err">)=</span><span class="mf">7.31</span><span class="err">ms</span><span class="w"> </span><span class="err">p(</span><span class="mi">95</span><span class="err">)=</span><span class="mf">99.49</span><span class="err">ms</span><span class="w">
</span><span class="p">{</span><span class="w"> </span><span class="err">expected_response:</span><span class="kc">true</span><span class="w"> </span><span class="p">}</span><span class="err">...:</span><span class="w"> </span><span class="err">avg=</span><span class="mi">8</span><span class="err">ms</span><span class="w"> </span><span class="err">min=</span><span class="mf">320.42</span><span class="err">µs</span><span class="w"> </span><span class="err">med=</span><span class="mf">1.52</span><span class="err">ms</span><span class="w"> </span><span class="err">max=</span><span class="mf">54.87</span><span class="err">s</span><span class="w"> </span><span class="err">p(</span><span class="mi">90</span><span class="err">)=</span><span class="mf">5.22</span><span class="err">ms</span><span class="w"> </span><span class="err">p(</span><span class="mi">95</span><span class="err">)=</span><span class="mf">7.86</span><span class="err">ms</span><span class="w">
</span><span class="err">✗</span><span class="w"> </span><span class="err">http_req_failed................:</span><span class="w"> </span><span class="mf">20.35</span><span class="err">%</span><span class="w"> </span><span class="err">✓</span><span class="w"> </span><span class="mi">45003</span><span class="w"> </span><span class="err">✗</span><span class="w"> </span><span class="mi">176055</span><span class="w">
</span><span class="err">http_req_receiving.............:</span><span class="w"> </span><span class="err">avg=</span><span class="mf">48.46</span><span class="err">µs</span><span class="w"> </span><span class="err">min=</span><span class="mi">0</span><span class="err">s</span><span class="w"> </span><span class="err">med=</span><span class="mf">22.25</span><span class="err">µs</span><span class="w"> </span><span class="err">max=</span><span class="mf">103.96</span><span class="err">ms</span><span class="w"> </span><span class="err">p(</span><span class="mi">90</span><span class="err">)=</span><span class="mf">54.67</span><span class="err">µs</span><span class="w"> </span><span class="err">p(</span><span class="mi">95</span><span class="err">)=</span><span class="mf">66.8</span><span class="err">µs</span><span class="w">
</span><span class="err">http_req_sending...............:</span><span class="w"> </span><span class="err">avg=</span><span class="mf">30.5</span><span class="err">µs</span><span class="w"> </span><span class="err">min=</span><span class="mi">0</span><span class="err">s</span><span class="w"> </span><span class="err">med=</span><span class="mf">26.5</span><span class="err">µs</span><span class="w"> </span><span class="err">max=</span><span class="mf">17.16</span><span class="err">ms</span><span class="w"> </span><span class="err">p(</span><span class="mi">90</span><span class="err">)=</span><span class="mf">48.81</span><span class="err">µs</span><span class="w"> </span><span class="err">p(</span><span class="mi">95</span><span class="err">)=</span><span class="mf">57.68</span><span class="err">µs</span><span class="w">
</span><span class="err">http_req_tls_handshaking.......:</span><span class="w"> </span><span class="err">avg=</span><span class="mf">12.72</span><span class="err">ms</span><span class="w"> </span><span class="err">min=</span><span class="mi">0</span><span class="err">s</span><span class="w"> </span><span class="err">med=</span><span class="mi">0</span><span class="err">s</span><span class="w"> </span><span class="err">max=</span><span class="mf">21.67</span><span class="err">s</span><span class="w"> </span><span class="err">p(</span><span class="mi">90</span><span class="err">)=</span><span class="mi">0</span><span class="err">s</span><span class="w"> </span><span class="err">p(</span><span class="mi">95</span><span class="err">)=</span><span class="mi">0</span><span class="err">s</span><span class="w">
</span><span class="err">http_req_waiting...............:</span><span class="w"> </span><span class="err">avg=</span><span class="mf">1.25</span><span class="err">s</span><span class="w"> </span><span class="err">min=</span><span class="mi">0</span><span class="err">s</span><span class="w"> </span><span class="err">med=</span><span class="mf">1.47</span><span class="err">ms</span><span class="w"> </span><span class="err">max=</span><span class="mi">1</span><span class="err">m</span><span class="mi">0</span><span class="err">s</span><span class="w"> </span><span class="err">p(</span><span class="mi">90</span><span class="err">)=</span><span class="mf">7.18</span><span class="err">ms</span><span class="w"> </span><span class="err">p(</span><span class="mi">95</span><span class="err">)=</span><span class="mf">99.4</span><span class="err">ms</span><span class="w">
</span><span class="err">http_reqs......................:</span><span class="w"> </span><span class="mi">221058</span><span class="w"> </span><span class="mf">669.854297</span><span class="err">/s</span><span class="w">
</span><span class="err">iteration_duration.............:</span><span class="w"> </span><span class="err">avg=</span><span class="mi">1</span><span class="err">m</span><span class="mi">11</span><span class="err">s</span><span class="w"> </span><span class="err">min=</span><span class="mf">27.92</span><span class="err">ms</span><span class="w"> </span><span class="err">med=</span><span class="mf">50.11</span><span class="err">s</span><span class="w"> </span><span class="err">max=</span><span class="mi">5</span><span class="err">m</span><span class="mi">26</span><span class="err">s</span><span class="w"> </span><span class="err">p(</span><span class="mi">90</span><span class="err">)=</span><span class="mi">2</span><span class="err">m</span><span class="mi">55</span><span class="err">s</span><span class="w"> </span><span class="err">p(</span><span class="mi">95</span><span class="err">)=</span><span class="mi">3</span><span class="err">m</span><span class="mi">52</span><span class="err">s</span><span class="w">
</span><span class="err">iterations.....................:</span><span class="w"> </span><span class="mi">5129</span><span class="w"> </span><span class="mf">15.541997</span><span class="err">/s</span><span class="w">
</span><span class="err">vus............................:</span><span class="w"> </span><span class="mi">1606</span><span class="w"> </span><span class="err">min=</span><span class="mi">1606</span><span class="w"> </span><span class="err">max=</span><span class="mi">2000</span><span class="w">
</span><span class="err">vus_max........................:</span><span class="w"> </span><span class="mi">2000</span><span class="w"> </span><span class="err">min=</span><span class="mi">2000</span><span class="w"> </span><span class="err">max=</span><span class="mi">2000</span><span class="w">
</span></code></pre></div></div>
<p>This tells us is that 221K requests were performed in 5m and 30s at an average rate of 670/s of which 20% failed, probably due to servers hitting capacity limits. The average request duration was over 1s but 95% of requests were served within 100ms.</p>
<h2 id="in-depth-statistics-and-visualisations">In-depth statistics and visualisations</h2>
<p>Having a short ASCII summary of a single test is cute, but that doesn’t tell us what we’re after. We need to know what happens to our machines, to our cluster, as we scale it up and… as it breaks. If it does, we need to know <em>how</em> it breaks, figure out <em>why</em>, remediate it and <em>confirm</em> that in fact we did.</p>
<p>In order to do that, we got <a href="https://k6.io/docs/results-output/real-time/influxdb-grafana/">k6 to write metrics to InfluxDB</a> and created a dashboard visualising the results in Grafana. Both of which we had set up prior to this scale out to investigate latency issues, as discussed in our <a href="https://blog.ipfs-search.com/challenge-accepted/">previous post</a>.</p>
<figure>
<img alt="Overview of our benchmarking dashboard." src="/assets/images/2023-04-18-searching-at-scale/Untitled%202.png" />
<figcaption>Overview of our benchmarking dashboard.</figcaption>
</figure>
<figure>
<img alt="This is what it looks like when we hit peak capacity." src="/assets/images/2023-04-18-searching-at-scale/Untitled%203.png" />
<figcaption>This is what it looks like when we hit peak capacity.</figcaption>
</figure>
<figure>
<img alt="It is often the maxing out of CPU on of 1 or 2 servers which casues the entire cluster to take increasingly longer lunches." src="/assets/images/2023-04-18-searching-at-scale/Untitled%204.png" />
<figcaption>
It is often the maxing out of CPU on of 1 or 2 servers which casues the entire cluster to take <a href="https://hitchhikers.fandom.com/wiki/Lig_Lury_Jr">increasingly longer lunches</a>.
</figcaption>
</figure>
<h1 id="not-the-scaling-we-expected">Not the scaling we expected</h1>
<p>As soon as we had the tests set up, we started plugging in servers. Over the past year we had been improving our <a href="https://github.com/ipfs-search/ipfs-search-deployment/">Ansible deployment stack</a> to be able to fully automatically install, configure and setup <a href="https://www.hetzner.com/dedicated-rootserver/matrix-ax">Hetzner bare metal</a> boxes, so we could deploy any number of nodes in about 30m.</p>
<figure>
<img src="/assets/images/2023-04-18-searching-at-scale/Untitled%205.png" />
<figcaption>Overview of all the (cold, with cleared frontend cache) benchmarks we've performed.</figcaption>
</figure>
<p>However, as we added nodes and thus capacity, we observed not only that the number of requests per second did not go up, the actual peak duration skyrocketed!</p>
<p>Specfically, with 33 nodes we were peaking around 700 RPS with a peak request duration of around 900ms. With 42 nodes we hit 750 RPS at about 1s. At 59 nodes we were again around 700 RPS with over 3s request durations. Something was definitely wrong!</p>
<p>As you may notice from the screenshot, we tried any number of tweaking of settings, upgrading OpenSearch, tweaking our API and even reinstalling our servers. One key aspect which kept returning is that the same 5 or so nodes were handling about 10x the IOPS of the other nodes. It turned out that somehow the cluster decided that these 5 nodes (despite or perhaps due to our myriad of shards) were handling a much greater share of the traffic and were causing a bottleneck in our cluster.</p>
<figure>
<img alt="IOPS in progress for all of our nodes. This is an indicator of the degree to which IO exhaustion is a bottleneck, particularly on NVMe-based setups (like ours). Note how most nodes are not even mentioned here, few have ~10 IOPS in progress and then there’s a few with ~100 in progress." src="/assets/images/2023-04-18-searching-at-scale/Untitled%206.png" />
<figcaption>IOPS in progress for all of our nodes. This is an indicator of the degree to which IO exhaustion is a bottleneck, particularly on NVMe-based setups (like ours). Note how most nodes are not even mentioned here, few have ~10 IOPS in progress and then there’s a few with ~100 in progress.</figcaption>
</figure>
<p>By this time, we had been trying for well over a year to meet our 1000 hit/s benchmark. By now, we really expected to have met the mark, simply by plugging in more servers. Yet, we were forced to acknowledge that a much deeper overhaul was necessary.</p>
<h2 id="we-get-by-with-a-little-help-from-our-friends">We get by with a little help from our friends</h2>
<p>By this point, we were despairing and decided to ask for help. Thus far, we had been assisted by <a href="https://dataforest.ai/">DataForest</a> for practical assistance in our deployment setup, like installing Grafana and migrating to OpenSearch. Although we had no budget left, they decided to help us and use our particular and (apparently rare) problem as a study case. They gave us extended and concrete recommendations on how to further optimize our cluster. We owe them a great debt of gratitude and respect, especially considering that they operate from a country in war, Ukraine.</p>
<p>In addition, we opted for a free trial with <a href="https://opster.com/">Opster</a> and, despite our honesty about limited budget they volunteered to have an in-depth look into our issues. They too, were quite suprised by our cluster’s odd behaviour, allocating so much load to just few of the nodes. It might not entirely be an accident that they published an article ‘<a href="https://opster.com/guides/opensearch/opensearch-operations/opensearch-hotspots/">OpenSearch Hotspots – Load Balancing, Data Allocation and How to Avoid Hotspots</a>’ shortly after assisting us… Regardless, we can’t express enough gratitude for the amount of real-world knowledge about Elastic and OpenSearch they freely put out there.</p>
<p>Both of these parties gave us roughly similar recommendations, among which:</p>
<ul>
<li>Put similar data in the same index.</li>
<li>Only index fields which you’re using.</li>
</ul>
<p>It did not lead to a single root cause. It seems our problem did not, in fact, have a single clear solution. A basic fact about <a href="https://en.wikipedia.org/wiki/Complex_system">complex systems</a>, confirmed.</p>
<h1 id="rethinking-our-index">Rethinking our index</h1>
<p>It became clear that we had to dig deeper. Despite despairing we could not give up, not with the amount of time already invested. A challenge is not a great challenge if one can be certain to make it!</p>
<h2 id="splitting-our-index">Splitting our index</h2>
<p>Not sure what caused our problems, we took a wild gamble and we decided to do what we had been postponing for years: splitting our index! This is like open brain surgery for search engines! It literally affects every single part of our stack.</p>
<p>We were certain that having 4 huge indexes (files, directories, invalids and partials) was not an optimal solution. We were sure it was going to give performance improvements of <em>some</em> kind. But it is truly challenging to re-index the close to 800 million documents. Just a single typo, and you’d have to do it again. Just a small coding mistake, and you’re losing data. Just one of 73 servers crashing, and you can start again.</p>
<h3 id="categorising-documents">Categorising documents</h3>
<p>And not just that… how exactly are we going to group our documents? Documents, audio, images, videos, directories and ‘other’, like we have in our <a href="https://ipfs-search.com/">frontend</a>? But what, on Earth’s name is the definition of a ‘document’!?</p>
<figure>
<img alt="List of categories in our frontend." src="/assets/images/2023-04-18-searching-at-scale/Untitled%207.png" />
<figcaption>List of categories in our frontend.</figcaption>
</figure>
<p>In order to make informed decisions about this, we decided to query our dataset for statistics, based on our ‘working’ definition of content types from the frontend. How many items of each category did we have? What sort of fields were present for various types and categories?</p>
<h3 id="field-statistics">Field statistics</h3>
<p>Hence, we produced <a href="https://github.com/ipfs-search/ipfs-search/tree/mapping_v10/docs/indices/fields">extensive statistics</a> with using <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics-scripted-metric-aggregation.html">scripted metrics</a>, as that’s the only way to gather statistics on unindexed fields:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// Init</span>
<span class="n">state</span><span class="o">.</span><span class="na">fields</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">HashMap</span><span class="o">();</span>
<span class="c1">// Map</span>
<span class="kt">void</span> <span class="nf">iterateHashMap</span><span class="o">(</span><span class="nc">String</span> <span class="n">prefix</span><span class="o">,</span> <span class="nc">HashMap</span> <span class="n">input</span><span class="o">,</span> <span class="nc">HashMap</span> <span class="n">output</span><span class="o">)</span> <span class="o">{</span>
<span class="n">input</span><span class="o">.</span><span class="na">forEach</span><span class="o">((</span><span class="n">key</span><span class="o">,</span> <span class="n">value</span><span class="o">)</span> <span class="o">-></span> <span class="o">{</span>
<span class="nc">String</span> <span class="n">fieldName</span> <span class="o">=</span> <span class="n">prefix</span> <span class="o">+</span> <span class="n">key</span><span class="o">;</span>
<span class="k">if</span> <span class="o">(</span><span class="n">value</span> <span class="k">instanceof</span> <span class="nc">Map</span><span class="o">)</span> <span class="o">{</span>
<span class="n">iterateHashMap</span><span class="o">(</span><span class="n">fieldName</span> <span class="o">+</span> <span class="sc">'.'</span><span class="o">,</span> <span class="n">value</span><span class="o">,</span> <span class="n">output</span><span class="o">);</span>
<span class="k">return</span> <span class="kc">null</span><span class="o">;</span>
<span class="o">}</span>
<span class="k">if</span> <span class="o">(</span><span class="n">output</span><span class="o">.</span><span class="na">containsKey</span><span class="o">(</span><span class="n">fieldName</span><span class="o">))</span> <span class="o">{</span>
<span class="n">output</span><span class="o">[</span><span class="n">fieldName</span><span class="o">]</span> <span class="o">+=</span> <span class="mi">1</span><span class="o">;</span>
<span class="o">}</span> <span class="k">else</span> <span class="o">{</span>
<span class="n">output</span><span class="o">[</span><span class="n">fieldName</span><span class="o">]</span> <span class="o">=</span> <span class="mi">1</span><span class="o">;</span>
<span class="o">}</span>
<span class="o">});</span>
<span class="o">}</span>
<span class="n">iterateHashMap</span><span class="o">(</span><span class="err">''</span><span class="o">,</span> <span class="n">params</span><span class="o">[</span><span class="err">'</span><span class="n">_source</span><span class="err">'</span><span class="o">],</span> <span class="n">state</span><span class="o">.</span><span class="na">fields</span><span class="o">);</span>
<span class="c1">// Combine</span>
<span class="n">state</span><span class="o">.</span><span class="na">fields</span>
<span class="c1">// Reduce</span>
<span class="nc">HashMap</span> <span class="n">output</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">HashMap</span><span class="o">();</span>
<span class="n">states</span><span class="o">.</span><span class="na">forEach</span><span class="o">(</span><span class="n">field</span> <span class="o">-></span> <span class="o">{</span>
<span class="n">field</span><span class="o">.</span><span class="na">forEach</span><span class="o">((</span><span class="n">fieldName</span><span class="o">,</span> <span class="n">count</span><span class="o">)</span> <span class="o">-></span> <span class="o">{</span>
<span class="k">if</span> <span class="o">(</span><span class="n">output</span><span class="o">.</span><span class="na">containsKey</span><span class="o">(</span><span class="n">fieldName</span><span class="o">))</span> <span class="o">{</span>
<span class="n">output</span><span class="o">[</span><span class="n">fieldName</span><span class="o">]</span> <span class="o">+=</span> <span class="n">count</span><span class="o">;</span>
<span class="o">}</span> <span class="k">else</span> <span class="o">{</span>
<span class="n">output</span><span class="o">[</span><span class="n">fieldName</span><span class="o">]</span> <span class="o">=</span> <span class="n">count</span><span class="o">;</span>
<span class="o">}</span>
<span class="o">})</span>
<span class="o">});</span>
<span class="k">return</span> <span class="n">output</span><span class="o">;</span>
</code></pre></div></div>
<p>Which adds up to the the following OpenSearch DSL query to get a list of fieldnames with occurance counts:</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="nl">"query"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"match_all"</span><span class="p">:</span><span class="w"> </span><span class="p">{}</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"size"</span><span class="p">:</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w">
</span><span class="nl">"aggs"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"aggs"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"scripted_metric"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"init_script"</span><span class="p">:</span><span class="w"> </span><span class="s2">"state.fields = new HashMap();"</span><span class="p">,</span><span class="w">
</span><span class="nl">"map_script"</span><span class="p">:</span><span class="w"> </span><span class="s2">"void iterateHashMap(String prefix, HashMap input, HashMap output) { for (entry in input.entrySet()) { String fieldName = prefix + entry.getKey(); if (entry.getValue() instanceof Map) { iterateHashMap(fieldName + '.', entry.getValue(), output); } else { if (output.containsKey(fieldName)) { output[fieldName] += 1; } else { output[fieldName] = 1; } } }}iterateHashMap('', params['_source'], state.fields);"</span><span class="p">,</span><span class="w">
</span><span class="nl">"combine_script"</span><span class="p">:</span><span class="w"> </span><span class="s2">"state.fields"</span><span class="p">,</span><span class="w">
</span><span class="nl">"reduce_script"</span><span class="p">:</span><span class="w"> </span><span class="s2">"HashMap output = new HashMap();for (fields in states) { for (field in fields.entrySet()) { String fieldName = field.getKey(); Integer count = field.getValue(); if (output.containsKey(fieldName)) { output[fieldName] += count; } else { output[fieldName] = count; } }}return output;"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>The result was a staggering amount of information, which we proceeded to sort out. We categorised each and every field: should it be copied, removed if it’s a duplicate, or simply not be indexed at all?</p>
<figure>
<img alt="Field statistics per type." src="/assets/images/2023-04-18-searching-at-scale/Untitled%208.png" />
<figcaption>Field statistics per type. <a href="https://github.com/ipfs-search/ipfs-search/blob/mapping_v10/docs/indices/sharding/content-types.pdf">Full dataset</a>.</figcaption>
</figure>
<figure>
<img alt="Document count per data type." src="/assets/images/2023-04-18-searching-at-scale/Untitled%209.png" />
<figcaption>Document count per data type. <a href="https://github.com/ipfs-search/ipfs-search/blob/mapping_v10/docs/indices/sharding/content-types.pdf">Full dataset</a>.</figcaption>
</figure>
<figure>
<img alt="Mime types in our index." src="/assets/images/2023-04-18-searching-at-scale/Untitled%2010.png" />
<figcaption>Mime types in our index. <a href="https://github.com/ipfs-search/ipfs-search/blob/mapping_v10/docs/indices/sharding/content-types.pdf">Full dataset</a>.</figcaption>
</figure>
<h3 id="mapping-all-the-things">Mapping All the Things</h3>
<p>With painstaking work and difficult decisions, we finally managed to arrive at a suitable mapping <a href="https://github.com/ipfs-search/ipfs-search/blob/mapping_v10/docs/indices/sharding/content-types.pdf">from mime types to indexes</a> as well as <a href="https://github.com/ipfs-search/ipfs-search/blob/mapping_v10/docs/indices/sharding/field_stats.pdf">which fields to index and how</a>. Specifically, for some fields we chose to copy the data to another field but retain the source document intact. For others, we simply eliminated the source field (see ‘Deduplicating fields’ below).</p>
<p>This resulted in monsters of mappings, such as the following (for documents):</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="nl">"dynamic"</span><span class="p">:</span><span class="w"> </span><span class="s2">"strict"</span><span class="p">,</span><span class="w">
</span><span class="nl">"properties"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"cid"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"keyword"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"content"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"text"</span><span class="p">,</span><span class="w">
</span><span class="nl">"term_vector"</span><span class="p">:</span><span class="w"> </span><span class="s2">"with_positions_offsets"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"content:character-count"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"integer"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"first-seen"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"date"</span><span class="p">,</span><span class="w">
</span><span class="nl">"format"</span><span class="p">:</span><span class="w"> </span><span class="s2">"strict_date_time"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"last-seen"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"date"</span><span class="p">,</span><span class="w">
</span><span class="nl">"format"</span><span class="p">:</span><span class="w"> </span><span class="s2">"strict_date_time"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"ipfs_tika_version"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"index"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"keyword"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"language"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"properties"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"confidence"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"index"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w">
</span><span class="nl">"doc_values"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"keyword"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"language"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"keyword"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"rawScore"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"double"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"references"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"properties"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"hash"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"keyword"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"name"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"text"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"parent_hash"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"keyword"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"size"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"long"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"urls"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"enabled"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"object"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"metadata"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"dynamic"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w">
</span><span class="nl">"properties"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"Content-Type"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"keyword"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"mime:type"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"keyword"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"mime:subtype"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"keyword"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"X-TIKA:Parsed-By"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"index"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"keyword"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"dc:title"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"text"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"dc:creator"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"text"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"dc:contributor"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"text"</span><span class="p">,</span><span class="w">
</span><span class="nl">"index"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w">
</span><span class="nl">"doc_values"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w">
</span><span class="nl">"copy_to"</span><span class="p">:</span><span class="w"> </span><span class="s2">"metadata.dc:creator"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"meta:last-author"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"text"</span><span class="p">,</span><span class="w">
</span><span class="nl">"index"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w">
</span><span class="nl">"doc_values"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w">
</span><span class="nl">"copy_to"</span><span class="p">:</span><span class="w"> </span><span class="s2">"metadata.dc:creator"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"article:author"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"text"</span><span class="p">,</span><span class="w">
</span><span class="nl">"index"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w">
</span><span class="nl">"doc_values"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w">
</span><span class="nl">"copy_to"</span><span class="p">:</span><span class="w"> </span><span class="s2">"metadata.dc:creator"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"dc:identifier"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"keyword"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"xmpMM:DocumentID"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"keyword"</span><span class="p">,</span><span class="w">
</span><span class="nl">"index"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w">
</span><span class="nl">"doc_values"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w">
</span><span class="nl">"copy_to"</span><span class="p">:</span><span class="w"> </span><span class="s2">"metadata.dc:identifier"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"xmpMM:DerivedFrom:DocumentID"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"keyword"</span><span class="p">,</span><span class="w">
</span><span class="nl">"index"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w">
</span><span class="nl">"doc_values"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w">
</span><span class="nl">"copy_to"</span><span class="p">:</span><span class="w"> </span><span class="s2">"metadata.dc:identifier"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"xmpMM:DerivedFrom:InstanceID"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"keyword"</span><span class="p">,</span><span class="w">
</span><span class="nl">"index"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w">
</span><span class="nl">"doc_values"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w">
</span><span class="nl">"copy_to"</span><span class="p">:</span><span class="w"> </span><span class="s2">"metadata.dc:identifier"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"Content Identifier"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"keyword"</span><span class="p">,</span><span class="w">
</span><span class="nl">"index"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w">
</span><span class="nl">"doc_values"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w">
</span><span class="nl">"copy_to"</span><span class="p">:</span><span class="w"> </span><span class="s2">"metadata.dc:identifier"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"dc:language"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"keyword"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"dc:description"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"text"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"dc:subject"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"text"</span><span class="p">,</span><span class="w">
</span><span class="nl">"copy_to"</span><span class="p">:</span><span class="w"> </span><span class="s2">"metadata.dc:description"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"meta:keyword"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"text"</span><span class="p">,</span><span class="w">
</span><span class="nl">"copy_to"</span><span class="p">:</span><span class="w"> </span><span class="s2">"metadata.dc:description"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"dc:publisher"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"keyword"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"dcterms:created"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"index"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"date"</span><span class="p">,</span><span class="w">
</span><span class="nl">"format"</span><span class="p">:</span><span class="w"> </span><span class="s2">"date_optional_time"</span><span class="p">,</span><span class="w">
</span><span class="nl">"ignore_malformed"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"dcterms:modified"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"index"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"date"</span><span class="p">,</span><span class="w">
</span><span class="nl">"format"</span><span class="p">:</span><span class="w"> </span><span class="s2">"date_optional_time"</span><span class="p">,</span><span class="w">
</span><span class="nl">"ignore_malformed"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"w:comments"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"text"</span><span class="p">,</span><span class="w">
</span><span class="nl">"index"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w">
</span><span class="nl">"doc_values"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w">
</span><span class="nl">"copy_to"</span><span class="p">:</span><span class="w"> </span><span class="s2">"content"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"xmpTPg:NPages"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"short"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"og:site_name"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"text"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"og_type"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"keyword"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"doi"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"keyword"</span><span class="p">,</span><span class="w">
</span><span class="nl">"index"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w">
</span><span class="nl">"doc_values"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w">
</span><span class="nl">"copy_to"</span><span class="p">:</span><span class="w"> </span><span class="s2">"metadata.dc:identifier"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"pdf:docinfo:custom:IEEE Publication ID"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"keyword"</span><span class="p">,</span><span class="w">
</span><span class="nl">"index"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w">
</span><span class="nl">"doc_values"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w">
</span><span class="nl">"copy_to"</span><span class="p">:</span><span class="w"> </span><span class="s2">"metadata.dc:identifier"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"pdf:docinfo:custom:IEEE Issue ID"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"keyword"</span><span class="p">,</span><span class="w">
</span><span class="nl">"index"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w">
</span><span class="nl">"doc_values"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w">
</span><span class="nl">"copy_to"</span><span class="p">:</span><span class="w"> </span><span class="s2">"metadata.dc:identifier"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"pdf:docinfo:custom:IEEE Article ID"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"keyword"</span><span class="p">,</span><span class="w">
</span><span class="nl">"index"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w">
</span><span class="nl">"doc_values"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w">
</span><span class="nl">"copy_to"</span><span class="p">:</span><span class="w"> </span><span class="s2">"metadata.dc:identifier"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"WPS-JOURNALDOI"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"keyword"</span><span class="p">,</span><span class="w">
</span><span class="nl">"index"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w">
</span><span class="nl">"doc_values"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w">
</span><span class="nl">"copy_to"</span><span class="p">:</span><span class="w"> </span><span class="s2">"metadata.dc:identifier"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>The final mappings for other types will soon be published in our <a href="https://ipfs-search.readthedocs.io/en/latest/indices/README.html">docs</a>. Until then, you may have a look at our <a href="https://github.com/ipfs-search/ipfs-search/tree/mapping_v10/docs/indices/v11%20WIP">WIP branch</a>.</p>
<p>In the end, we indexed the following types together:</p>
<ul>
<li>(Compressed) archives: ZIP, tarballs, etc.</li>
<li>(Textual) Documents: Text, HTML, Word, PDF, PowerPoint, etc.</li>
<li>Images</li>
<li>Videos</li>
<li>Audio</li>
<li>Directories</li>
<li>Data: JSON, binary blobs, etc.</li>
<li>Unknown: files with no extracted metadata whatsoever</li>
<li>Other: Anything not in any of the aforementioned categories</li>
</ul>
<h2 id="data-cleanup">Data cleanup</h2>
<h3 id="hashing-out-12-billion-links">Hashing out 12 billion links!</h3>
<p>During earlier development on the crawler, the suspicion started to arise that some of the documents in our index were ‘slightly’ larger than others. While implementing Redis caching for our indexer we discovered that some documents had thousands of links to them, whereas most documents have just one or a few.</p>
<p>It wasn’t until we wrote our <a href="https://github.com/ipfs-search/ipfs-search-linksplitter/tree/main">linksplitter</a> that we fully realized how bad the problem was! And in hindsight, it makes perfect sense: millions of Wikipedia articles with regular updates create new references to the same documents all the time. So we discovered that without knowing it, we were searching through a whopping 12.193.745.087 links! No wonder our search was slow!</p>
<p>At this time we merely split out the links to optimise search performance (chucking any but the last 8 references away during reindexing). But like you, we can’t wait to bring this dataset to the world — preferably loading it into and serving it from an actual graph database.</p>
<p><strong><em>We have a graph of links on IPFS going back to 2019!</em></strong></p>
<figure>
<img alt="I couldn’t help but exploring a little bit what (a tiny fraction of) IPFS’ content graph looks like." src="/assets/images/2023-04-18-searching-at-scale/Untitled%2011.png" />
<figcaption>I couldn’t help but exploring a little bit what (a tiny fraction of) IPFS’ content graph looks like.</figcaption>
</figure>
<h3 id="deduplicating-fields">Deduplicating fields</h3>
<p>Many fields were duplicates due to a lack of clear metadata standards in our metadata extractor based on <a href="https://tika.apache.org/">Apache’s Tika</a>. This meant carefully looking at our data, it meant scrutinizing Apache Tika’s <a href="https://cwiki.apache.org/confluence/display/TIKA/Migrating+to+Tika+2.0.0">developer documentation</a> on metadata keys and it meant writing an intimidating ‘Painless’ (Elastic/OpenSearch <a href="https://www.elastic.co/guide/en/elasticsearch/reference/master/modules-scripting-painless.html">built-in scripting language</a>) to <a href="https://github.com/ipfs-search/ipfs-search/blob/mapping_v10/docs/indices/painless/harmonize_values.painless">harmonize fields</a>. In the process, we also created a shell-script <a href="https://github.com/ipfs-search/ipfs-search/blob/mapping_v10/docs/indices/painless/upload_painless.sh">‘Painless’ uploader</a> to make uploading ‘painless’ less … painful.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kt">void</span> <span class="nf">harmonizeField</span><span class="o">(</span><span class="nc">HashMap</span> <span class="n">ctx</span><span class="o">,</span> <span class="nc">String</span> <span class="n">srcFieldName</span><span class="o">,</span> <span class="nc">String</span> <span class="n">dstFieldName</span><span class="o">)</span> <span class="o">{</span>
<span class="k">if</span> <span class="o">(</span><span class="n">ctx</span><span class="o">.</span><span class="na">containsKey</span><span class="o">(</span><span class="n">srcFieldName</span><span class="o">))</span> <span class="o">{</span>
<span class="nc">ArrayList</span> <span class="n">srcValues</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">[</span><span class="n">srcFieldName</span><span class="o">];</span>
<span class="k">if</span> <span class="o">(</span><span class="n">ctx</span><span class="o">.</span><span class="na">containsKey</span><span class="o">(</span><span class="n">dstFieldName</span><span class="o">))</span> <span class="o">{</span>
<span class="nc">ArrayList</span> <span class="n">dstValues</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">[</span><span class="n">dstFieldName</span><span class="o">];</span>
<span class="k">if</span> <span class="o">(</span><span class="n">srcValues</span> <span class="o">==</span> <span class="n">dstValues</span><span class="o">)</span> <span class="o">{</span>
<span class="c1">// src and dst values are equal, remove src</span>
<span class="n">ctx</span><span class="o">.</span><span class="na">remove</span><span class="o">(</span><span class="n">srcFieldName</span><span class="o">);</span>
<span class="o">}</span>
<span class="k">return</span><span class="o">;</span>
<span class="o">}</span>
<span class="n">ctx</span><span class="o">[</span><span class="n">dstFieldName</span><span class="o">]</span> <span class="o">=</span> <span class="n">srcValues</span><span class="o">;</span>
<span class="n">ctx</span><span class="o">.</span><span class="na">remove</span><span class="o">(</span><span class="n">srcFieldName</span><span class="o">);</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="nc">String</span> <span class="n">nestedKey</span> <span class="o">=</span> <span class="err">'</span><span class="n">metadata</span><span class="err">'</span><span class="o">;</span>
<span class="nc">Map</span> <span class="n">remapFields</span> <span class="o">=</span> <span class="o">[</span>
<span class="o">...</span>
<span class="err">'</span><span class="nl">w:</span><span class="n">comments</span><span class="err">'</span><span class="o">:</span> <span class="err">'</span><span class="nl">w:</span><span class="nc">Comments</span><span class="err">'</span><span class="o">,</span>
<span class="err">'</span><span class="n">comment</span><span class="err">'</span><span class="o">:</span> <span class="err">'</span><span class="nl">w:</span><span class="nc">Comments</span><span class="err">'</span><span class="o">,</span>
<span class="err">'</span><span class="nc">Comments</span><span class="err">'</span><span class="o">:</span> <span class="err">'</span><span class="nl">w:</span><span class="nc">Comments</span><span class="err">'</span><span class="o">,</span>
<span class="err">'</span><span class="no">JPEG</span> <span class="nc">Comment</span><span class="err">'</span><span class="o">:</span> <span class="err">'</span><span class="nl">w:</span><span class="nc">Comments</span><span class="err">'</span><span class="o">,</span>
<span class="err">'</span><span class="nc">Exif</span> <span class="nl">SubIFD:</span><span class="nc">User</span> <span class="nc">Comment</span><span class="err">'</span><span class="o">:</span> <span class="err">'</span><span class="nl">w:</span><span class="nc">Comments</span><span class="err">'</span><span class="o">,</span>
<span class="err">'</span><span class="nc">User</span> <span class="nc">Comment</span><span class="err">'</span><span class="o">:</span> <span class="err">'</span><span class="nl">w:</span><span class="nc">Comments</span><span class="err">'</span><span class="o">,</span>
<span class="o">...</span>
<span class="o">];</span>
<span class="k">if</span> <span class="o">(!</span><span class="n">ctx</span><span class="o">.</span><span class="na">containsKey</span><span class="o">(</span><span class="n">nestedKey</span><span class="o">))</span> <span class="k">return</span><span class="o">;</span>
<span class="nc">HashMap</span> <span class="n">nestedCtx</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">[</span><span class="n">nestedKey</span><span class="o">];</span>
<span class="k">if</span> <span class="o">(</span><span class="n">nestedCtx</span> <span class="o">==</span> <span class="kc">null</span><span class="o">)</span> <span class="k">return</span><span class="o">;</span>
<span class="k">for</span> <span class="o">(</span><span class="n">entry</span> <span class="n">in</span> <span class="n">remapFields</span><span class="o">.</span><span class="na">entrySet</span><span class="o">())</span> <span class="o">{</span>
<span class="nc">String</span> <span class="n">srcFieldName</span> <span class="o">=</span> <span class="n">entry</span><span class="o">.</span><span class="na">getKey</span><span class="o">();</span>
<span class="nc">String</span> <span class="n">dstFieldName</span> <span class="o">=</span> <span class="n">entry</span><span class="o">.</span><span class="na">getValue</span><span class="o">();</span>
<span class="n">harmonizeField</span><span class="o">(</span><span class="n">nestedCtx</span><span class="o">,</span> <span class="n">srcFieldName</span><span class="o">,</span> <span class="n">dstFieldName</span><span class="o">);</span>
<span class="o">}</span>
</code></pre></div></div>
<h3 id="re-hashing-document-ids">Re-hashing document ID’s</h3>
<p>During critical reflection along the lines of “why are these 5 darn servers taking all our load!??? 🤯”, we figured that one potential cause could be an unequal distributions of documents among shards due to us using <a href="https://docs.ipfs.tech/concepts/content-addressing/">IPFS/IPLD CID</a>’s as document identifiers. See, all CID’s start with just one of a few options of the same bytes.</p>
<p>If the underlying index doesn’t re-hash them, a lot of documents could end up together in ways which are sub-optimal and/or the shard distribution could end up all messed up. As we weren’t able to find conclusive evidence in OpenSearch’ code as to whether or not custom (vs. generated) DocID’s were hashed, we decided to re-hash them using SHA1.</p>
<p>As a bonus, this allowed us to do one other thing we’ve been wanting to do, which is adding a protocol identifier to our documents, paving the way for future support of other content-addressed protocols (e.g. <code class="language-plaintext highlighter-rouge">ipfs://bafy...</code> over<code class="language-plaintext highlighter-rouge">bafy...</code>).</p>
<p>Luckily, someone published a <a href="https://github.com/sektorcap/sha-painless/blob/master/sha1.painless">SHA1 implementation for Painless</a> which we thankfully made use of!</p>
<h3 id="other-painless-stuff">Other ‘Painless’ stuff</h3>
<p>While we were undertaking the huge effort of Reindexing All the Things, we decided to also implement some other ‘nice to haves’:</p>
<ul>
<li><a href="https://github.com/ipfs-search/ipfs-search/blob/mapping_v10/docs/indices/painless/crop_content.painless">Cropping body content to 1 MB</a> (some were as large as 10 MB!).</li>
<li><a href="https://github.com/ipfs-search/ipfs-search/blob/mapping_v10/docs/indices/painless/split_mime.painless">Splitting mime types</a> into their constituent type, subtype and parameters.</li>
<li><a href="https://github.com/ipfs-search/ipfs-search/blob/mapping_v10/docs/indices/painless/content_size.painless">Adding character counts (size)</a> for body content.</li>
</ul>
<h2 id="re-index-from-hell">Re-index From Hell</h2>
<p>With the mapping and all our scripts snugly fit into an <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest.html">ingest pipeline</a>, we were ready to start re-indexing. Writing horrendous queries such as this to sort our documents based on mimetype along the process:</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="nl">"source"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"index"</span><span class="p">:</span><span class="w"> </span><span class="s2">"ipfs_files_v9"</span><span class="p">,</span><span class="w">
</span><span class="nl">"query"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"bool"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"filter"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"range"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"first-seen"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"gte"</span><span class="p">:</span><span class="w"> </span><span class="mi">2023</span><span class="p">,</span><span class="w">
</span><span class="nl">"lt"</span><span class="p">:</span><span class="w"> </span><span class="mi">2024</span><span class="p">,</span><span class="w">
</span><span class="nl">"format"</span><span class="p">:</span><span class="w"> </span><span class="s2">"yyyy"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"should"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="p">{</span><span class="w"> </span><span class="nl">"wildcard"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"metadata.Content-Type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"text/x-web-markdown*"</span><span class="w"> </span><span class="p">}},</span><span class="w">
</span><span class="p">{</span><span class="w"> </span><span class="nl">"wildcard"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"metadata.Content-Type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"text/x-rst*"</span><span class="w"> </span><span class="p">}},</span><span class="w">
</span><span class="p">{</span><span class="w"> </span><span class="nl">"wildcard"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"metadata.Content-Type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"text/x-log*"</span><span class="w"> </span><span class="p">}},</span><span class="w">
</span><span class="p">{</span><span class="w"> </span><span class="nl">"wildcard"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"metadata.Content-Type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"text/x-asciidoc*"</span><span class="w"> </span><span class="p">}},</span><span class="w">
</span><span class="p">{</span><span class="w"> </span><span class="nl">"wildcard"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"metadata.Content-Type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"text/troff*"</span><span class="w"> </span><span class="p">}},</span><span class="w">
</span><span class="p">{</span><span class="w"> </span><span class="nl">"wildcard"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"metadata.Content-Type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"text/plain*"</span><span class="w"> </span><span class="p">}},</span><span class="w">
</span><span class="p">{</span><span class="w"> </span><span class="nl">"wildcard"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"metadata.Content-Type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"text/html*"</span><span class="w"> </span><span class="p">}},</span><span class="w">
</span><span class="p">{</span><span class="w"> </span><span class="nl">"wildcard"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"metadata.Content-Type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"message/rfc822*"</span><span class="w"> </span><span class="p">}},</span><span class="w">
</span><span class="p">{</span><span class="w"> </span><span class="nl">"wildcard"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"metadata.Content-Type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"message/news*"</span><span class="w"> </span><span class="p">}},</span><span class="w">
</span><span class="p">{</span><span class="w"> </span><span class="nl">"wildcard"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"metadata.Content-Type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"image/vnd.djvu*"</span><span class="w"> </span><span class="p">}},</span><span class="w">
</span><span class="p">{</span><span class="w"> </span><span class="nl">"wildcard"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"metadata.Content-Type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"application/xhtml+xml*"</span><span class="w"> </span><span class="p">}},</span><span class="w">
</span><span class="p">{</span><span class="w"> </span><span class="nl">"wildcard"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"metadata.Content-Type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"application/x-tika-ooxml*"</span><span class="w"> </span><span class="p">}},</span><span class="w">
</span><span class="p">{</span><span class="w"> </span><span class="nl">"wildcard"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"metadata.Content-Type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"application/x-tika-msoffice*"</span><span class="w"> </span><span class="p">}},</span><span class="w">
</span><span class="p">{</span><span class="w"> </span><span class="nl">"wildcard"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"metadata.Content-Type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"application/x-tex*"</span><span class="w"> </span><span class="p">}},</span><span class="w">
</span><span class="p">{</span><span class="w"> </span><span class="nl">"wildcard"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"metadata.Content-Type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"application/x-mobipocket-ebook*"</span><span class="w"> </span><span class="p">}},</span><span class="w">
</span><span class="p">{</span><span class="w"> </span><span class="nl">"wildcard"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"metadata.Content-Type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"application/x-fictionbook+xml*"</span><span class="w"> </span><span class="p">}},</span><span class="w">
</span><span class="p">{</span><span class="w"> </span><span class="nl">"wildcard"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"metadata.Content-Type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"application/x-dvi*"</span><span class="w"> </span><span class="p">}},</span><span class="w">
</span><span class="p">{</span><span class="w"> </span><span class="nl">"wildcard"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"metadata.Content-Type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"application/vnd.sun.xml.writer.global*"</span><span class="w"> </span><span class="p">}},</span><span class="w">
</span><span class="p">{</span><span class="w"> </span><span class="nl">"wildcard"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"metadata.Content-Type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"application/vnd.openxmlformats-officedocument.wordprocessingml.document*"</span><span class="w"> </span><span class="p">}},</span><span class="w">
</span><span class="p">{</span><span class="w"> </span><span class="nl">"wildcard"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"metadata.Content-Type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"application/vnd.openxmlformats-officedocument.spreadsheetml.sheet*"</span><span class="w"> </span><span class="p">}},</span><span class="w">
</span><span class="p">{</span><span class="w"> </span><span class="nl">"wildcard"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"metadata.Content-Type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"application/vnd.openxmlformats-officedocument.presentationml.presentation*"</span><span class="w"> </span><span class="p">}},</span><span class="w">
</span><span class="p">{</span><span class="w"> </span><span class="nl">"wildcard"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"metadata.Content-Type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"application/vnd.oasis.opendocument.text*"</span><span class="w"> </span><span class="p">}},</span><span class="w">
</span><span class="p">{</span><span class="w"> </span><span class="nl">"wildcard"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"metadata.Content-Type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"application/vnd.ms-powerpoint*"</span><span class="w"> </span><span class="p">}},</span><span class="w">
</span><span class="p">{</span><span class="w"> </span><span class="nl">"wildcard"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"metadata.Content-Type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"application/vnd.ms-htmlhelp*"</span><span class="w"> </span><span class="p">}},</span><span class="w">
</span><span class="p">{</span><span class="w"> </span><span class="nl">"wildcard"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"metadata.Content-Type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"application/vnd.ms-excel*"</span><span class="w"> </span><span class="p">}},</span><span class="w">
</span><span class="p">{</span><span class="w"> </span><span class="nl">"wildcard"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"metadata.Content-Type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"application/vnd.sun.xml.draw*"</span><span class="w"> </span><span class="p">}},</span><span class="w">
</span><span class="p">{</span><span class="w"> </span><span class="nl">"wildcard"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"metadata.Content-Type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"application/rtf*"</span><span class="w"> </span><span class="p">}},</span><span class="w">
</span><span class="p">{</span><span class="w"> </span><span class="nl">"wildcard"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"metadata.Content-Type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"application/postscript*"</span><span class="w"> </span><span class="p">}},</span><span class="w">
</span><span class="p">{</span><span class="w"> </span><span class="nl">"wildcard"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"metadata.Content-Type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"application/pdf*"</span><span class="w"> </span><span class="p">}},</span><span class="w">
</span><span class="p">{</span><span class="w"> </span><span class="nl">"wildcard"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"metadata.Content-Type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"application/msword5*"</span><span class="w"> </span><span class="p">}},</span><span class="w">
</span><span class="p">{</span><span class="w"> </span><span class="nl">"wildcard"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"metadata.Content-Type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"application/msword2*"</span><span class="w"> </span><span class="p">}},</span><span class="w">
</span><span class="p">{</span><span class="w"> </span><span class="nl">"wildcard"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"metadata.Content-Type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"application/msword*"</span><span class="w"> </span><span class="p">}},</span><span class="w">
</span><span class="p">{</span><span class="w"> </span><span class="nl">"wildcard"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"metadata.Content-Type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"application/epub+zip*"</span><span class="w"> </span><span class="p">}}</span><span class="w">
</span><span class="p">],</span><span class="w">
</span><span class="nl">"minimum_should_match"</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"dest"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"index"</span><span class="p">:</span><span class="w"> </span><span class="s2">""</span><span class="p">,</span><span class="w">
</span><span class="nl">"pipeline"</span><span class="p">:</span><span class="w"> </span><span class="s2">"ipfs_files_cleanup_v11"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<h3 id="year-by-year">Year by year</h3>
<p>This in and by itself was already a bit of a tedious process, but then we started having stability issues and operations started to unpredictably crash. With over 300 million documents to re-index (the others being invalids and partials, not requiring re-indexing) we couldn’t risk losing all of our progress so, as you see, we started indexing documents by year.</p>
<figure>
<img src="/assets/images/2023-04-18-searching-at-scale/Untitled%2012.png" />
</figure>
<h3 id="10-documents-at-a-time">10 documents at a time</h3>
<p>As you might have gathered from some of the subtle hints above, <em>some</em> of the documents in our index were really humongous in size. Some have well over 10 MB in links/references, some have up to 10 MB in body content (full text-indexed!). Knowing that OpenSearch’ indexing bulk indexing buffers are 100 MB, this turned out quite problematic.</p>
<p>While the point of our re-index is exactly to get rid of these huge documents, in order to do so we’d have to process them and, without jumping through even more horrible hoops, meant indexing nearly a billion documents in batches of 10, in order not to overflow Elastic’s buffer.</p>
<h3 id="out-of-file-descriptors">Out of file descriptors!?</h3>
<p>And then, during the process we increasingly experienced random nodes disappearing. By now, we were used to quite a bit of 💩 from OpenSearch. But it got to the point where we were literally unable to complete simple indexing operations without a node disappearing, killing the re-index <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/scroll-api.html">scroll</a> in the process.</p>
<p>We got these weird and uncommon exceptions from OpenSearch, telling us there were not enough <a href="https://www.baeldung.com/linux/limit-file-descriptors">file descriptors</a>. What the!?? So, we dug deeper…</p>
<p>It turned out that not OpenSearch. Not <a href="https://github.com/ipfs/kubo">Kubo</a>. Not our crawler but… <a href="https://www.influxdata.com/time-series-platform/telegraf/">Telegraf</a>, Influx’ metrics collecting daemon, was eating our file descriptors. And not just in any way, it was doing so tediously slowly, adding 1 FD per second, creating a problem which was so slow to emerge that it took months to manifest.</p>
<figure>
<img src="/assets/images/2023-04-18-searching-at-scale/Untitled%2013.png" />
</figure>
<p>Once it was discovered that Telegraf was the culprit though, it was easy enough to identify the <a href="https://github.com/ipfs-search/ipfs-search-deployment/commit/3cf83e30914e700d0e946922c04126f8f0fbd1e5#diff-6baffab2b19ba2a37c98f89821efdd34eff3117990b639d749e98fc2d18a8144R5279">malignant code</a>, a plugin logging ethernet statistics to attempt to diagnose the scaling behaviour discussed prior in this post (in order to exclude ethernet ring buffer overflows).</p>
<h3 id="open-source-is-awesome">Open Source is Awesome</h3>
<p>Being good FOSS citizens, an <a href="https://github.com/influxdata/telegraf/issues/12813">issue</a> was created on Influx’ Telegraf repo which was <em>reviewed that same day</em>. Only to find that the next day they already had a PR ready, complete with an artifact allowing us to verify that the issue was indeed resolved. Within 6 days Influx released an updated version.</p>
<figure>
<img src="/assets/images/2023-04-18-searching-at-scale/Untitled%2014.png" />
</figure>
<p>This is incredible! We ❤️ 💚 💙 💖 Open Source! And… great work <a href="https://www.influxdata.com/">InfluxDB</a>, you got good things going on! 👀</p>
<p>As soon as this was resolved, our cluster was rock stable again and <em>finally</em> managed to Reindex All the Things. Ready for testing!</p>
<h2 id="re-sharding-all-the-shards">Re-sharding All the Shards</h2>
<p>Except, not really. When you’re building an index in OpenSearch/ElasticSearch, you kind of have to guesstimate the amount of shards. The general recommendation is that a single shard should be between 10 and 50 GB in size, ideally 20-30 GB. Yet, there’s no reliable way to know the size of the index ahead of … indexing.</p>
<p>Of course we estimated the size of shards, using the fraction of total documents times the total size (~15 TB) of our files index. But as we discovered, many documents have different sizes, different fields and some of our estimates were way off.</p>
<p>Eventually, we had to <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-shrink-index.html">shrink</a> (merge shards) on some of our indexes and <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-split-index.html">split</a> shards on some others, finally we brought all of them within the desirable range. Mind you, our cluster only handles these kinds of processes well for one index at a time and they can take up to 24h to complete.</p>
<p>After which we set up replication and waited another half a day for the cluster to balance. And then, only then, are we ready to actually use our indexes.</p>
<h1 id="rewriting-our-api-server">Rewriting our API server</h1>
<p>But wait… we just went from searching 2 indexes (files and directories) to searching 9 of them! However we approach this, it means a profound change to the way our queries function. How are we going to integrate that into our <a href="https://github.com/ipfs-search/ipfs-search-api/tree/master/server">vanilla JS API server</a>, most of which has not been touched in a year? Particularly, how are we going to make sure that we’re not missing out on relevant search results just because we made a silly typo?</p>
<figure>
<img alt="We can only abuse a memo so many times without giving some credit…" src="/assets/images/2023-04-18-searching-at-scale/Untitled%2015.png" />
<figcaption>We can only abuse a memo so many times without giving some credit…</figcaption>
</figure>
<h2 id="typing-all-the-things">Typing All the Things!</h2>
<p>Our solution was to Rewrite it in <del>Rust</del> TypeScript. Simply put, we have a lot of literals, there is a lot of code to rewrite/migrate — our API server really hasn’t gotten the love it deserves, pending a full rewrite like this. Type inference in this case allows us to do abstract reasoning over types such that if our code isn’t right, it simply won’t compile.</p>
<figure>
<img src="/assets/images/2023-04-18-searching-at-scale/Untitled%2016.png" />
</figure>
<p>For example, in the API server we’ve created types for:</p>
<ul>
<li><a href="https://github.com/ipfs-search/ipfs-search-api/blob/rewrite/packages/server/src/search/documentfields.ts">Document fields</a>, with field name literals.</li>
<li><a href="https://github.com/ipfs-search/ipfs-search-api/blob/rewrite/packages/server/src/search/source.ts">Document’s source</a>, as a subset of document fields.</li>
<li><a href="https://github.com/ipfs-search/ipfs-search-api/blob/rewrite/packages/server/src/search/queryfields.ts">Query fields</a> (including boosts and highlights), again restrained by document fields.</li>
</ul>
<p>Thanks to this approach, it becomes literally impossible to refer to non-existing fields or missing data because a typo in a field name (except, of course, where the literals are defined). We already caught several bugs of which we were not previously aware from our older API code.</p>
<p>We also created a new common <a href="https://github.com/ipfs-search/ipfs-search-api/tree/rewrite/packages/types">types</a> library, we’ve implemented types for:</p>
<ul>
<li><a href="https://github.com/ipfs-search/ipfs-search-api/blob/rewrite/packages/types/src/searchquery.ts">Search queries</a>.</li>
<li><a href="https://github.com/ipfs-search/ipfs-search-api/blob/rewrite/packages/types/src/doctypes.ts">Document types</a>.</li>
<li><a href="https://github.com/ipfs-search/ipfs-search-api/blob/rewrite/packages/types/src/searchresult.ts">Search results</a>.</li>
</ul>
<p>The shared types between client and server allows for much stronger consistency in implementation. This will help us and you, power-user you are, to talk to our service in predictable and reliable ways.</p>
<h2 id="searching-for-subtypes-through-our-new-indexes">Searching for subtypes through our new indexes</h2>
<p>As a bonus of this great rewrite, users will soon have access to a <code class="language-plaintext highlighter-rouge">subtype</code> field in addition to <code class="language-plaintext highlighter-rouge">type</code> in queries and results, based on our newly generated indexes. This will have zero resource-impact for us (rather the opposite) and will allow you to query directly for:</p>
<ul>
<li>Archives.</li>
<li>Audio.</li>
<li>Data.</li>
<li>Documents.</li>
<li>Images.</li>
<li>Videos.</li>
<li>Unknown’s and;</li>
<li>the illustrious ‘Other’.</li>
</ul>
<h2 id="monorepos-for-js-hipsters-️">Monorepo’s for JS hipsters 〰️</h2>
<p>Like all the fashionable kids (and some of our <a href="https://research.google/pubs/pub45424/">Goliath competitors</a>) these days, we decided to rock with a proper monorepo with our client, our server and types as separate packages bundled snuggly together. To orchestrate it all, we opted for <a href="https://lerna.js.org/">Lerna</a>, the now-not-so-hip anymore wrapper around the increasingly hyped <a href="https://nx.dev/">Nx</a> build system.</p>
<figure>
<img alt="This guy’s using Lerna. He’s hip." src="/assets/images/2023-04-18-searching-at-scale/Untitled%2017.png" />
<figcaption>This guy’s using Lerna. He’s hip. (Shamelessly gleaned from <a href="http://www.slidedeck.io/Swiip/industrial-javascript">Matthieu Lux’s presentation</a>.)</figcaption>
</figure>
<p>This not only allowed us street cred’ and swag around places where espresso is served so fashionably bitter it turns your cheeks concave, it <em>also</em> allows us to:</p>
<ul>
<li>Publish everything to NPM at once.</li>
<li>Keeping versions in sync.</li>
<li>Perform end-to-end integration testing from client to server.</li>
<li>Rapidly iterate on the API, ensuring consistently without managing tons of repo’s.</li>
</ul>
<p>Isn’t JavaScript, I mean ECMAScript, I mean TypeScript, I mean Node, I mean NPM, native ESM, I mean ALL OF THIS JUNK TOOLING which gets replaced every 3 DOODLING MONTHS AMAZING? Hipster 💩, yes. We’re into it!</p>
<p>Now, we can do all the things Perl people were doing in the 90’s. Except, with <a href="https://prettier.io/">Prettier</a>, our code doesn’t look like <a href="https://en.wikipedia.org/wiki/Larry_Wall">Larry Wall</a> <a href="https://tumble.philadams.net/post/120728564/1987-larry-wall-falls-asleep-and-hits-larry">fell asleep on his keyboard</a> once.</p>
<p>Anyways, as with all our stuff, the <a href="https://github.com/ipfs-search/ipfs-search-api/tree/rewrite">Source is Out There</a>(tm) and soon, arguably, merged to main and published to NPM (which is not at all like <a href="https://www.cpan.org/">CPAN</a> and <em>definitely</em> not as well designed!).</p>
<figure>
<img alt="Larry Wall" src="https://upload.wikimedia.org/wikipedia/commons/b/b3/Larry_Wall_YAPC_2007.jpg" />
<figcaption><a href="https://en.wikipedia.org/wiki/Larry_Wall">Larry Wall</a> was the Original Hipster.</figcaption>
</figure>
<h2 id="ready-for-testing">Ready for testing!</h2>
<p>So now, without further ado, we are really ready for testing!</p>
<p>That <a href="https://www.goodreads.com/quotes/1398-i-love-deadlines-i-love-the-whooshing-noise-they-make">sound</a> you hear when the deadline’s passed by, yes, we heard it. A couple of times. Not to mention the sound of 💸 we could have made while we were making the first and only search engine for IPFS more awesome.</p>
<p>But, lo and behold…</p>
<h1 id="1300-hits-wow-uau-">1300 hit/s! Wow! Uau! 😮💥</h1>
<p>“Uau”, that’s what Portuguese people say. And it happens, I live there. So that’s what I said.</p>
<p>Remember that graph we started with? Noticed the part where with the same amount of nodes we suddently jumped up? And where request duration plummeted down? No, I am not making a reference to the disgraceful state of the climate or the economy. <em>There’s good things happening in this world.</em></p>
<p>Our search engine getting incredibly faster, for one thing. We hit well over 1300 hits per second, 30% more than we expected to, with only 75% of the 100 nodes we estimated for it. That’s equivalent of serving over 3000 users so fast they will not even know they were waiting.</p>
<p>Soon(tm), because although our goal it has succeeded, QED and all, there is still a bit of cleanup to do!</p>
<div style="display: flex; flex-wrap: nowrap;">
<figure style="width: 30vb; text-align: center;">
<img alt="Requests per second shooting up like El Niño off the coast of Peru." src="/assets/images/2023-04-18-searching-at-scale/Untitled%2018.png" width="164" style="width: 164px;" />
<figcaption>Requests per second shooting up like <a href="https://mobile.twitter.com/LeonSimons8/status/1646180075669209091">El Niño off the coast of Peru.</a></figcaption>
</figure>
<figure style="width: 30vb; text-align: center;">
<img alt="Request durations dropping like the value of Bored Apes." src="/assets/images/2023-04-18-searching-at-scale/Untitled%2019.png" width="116" style="width: 116px;" />
<figcaption>Request durations dropping <a href="https://www.glossy.co/fashion/2022-was-the-year-of-the-nft-reality-check/">like the value of Bored Apes</a> after bored rich monkeys realized they paid for the proof of having paid for something.</figcaption>
</figure>
</div>
<h2 id="wrapping-things-up">Wrapping things up</h2>
<p>For one, we are not yet indexing new stuff until we’ve refactored our <a href="https://www.notion.so/ipfs-search-com-roadmap-23-3ddab684f8ba4f14a3777dda893e1ed0">crawler</a>. Only then we can add what’s been indexed since we ran this test. And only then can we throw away our old index, making space, scaling down our cluster again to what we currently need. In the full awareness that…</p>
<h1 id="we-are-ready-for-it">We Are Ready For It!</h1>
<p>Bring it on! Users of the world, unite! Come, <a href="https://ipfs-search.com/">seek with us</a> the Interplanetary Filesystem and thou shalt <a href="https://ipfs-search.com/#/search/detail/video/QmPwRWz5mxvJDCk2d6MtXfZwYHPZqdDMoGHhkvsDN3o5Jh?q=gangnam&page=1&type=video&last_seen=Any">find</a>!</p>
<figure>
<a href="https://ipfs-search.com/#/search/detail/video/QmPwRWz5mxvJDCk2d6MtXfZwYHPZqdDMoGHhkvsDN3o5Jh?q=gangnam&page=1&type=video&last_seen=Any">
<img alt="We are ready!" src="/assets/images/2023-04-18-searching-at-scale/Untitled%2020.png" />
</a>
<figcaption>
<a href="https://ipfs-search.com/#/search/detail/video/QmPwRWz5mxvJDCk2d6MtXfZwYHPZqdDMoGHhkvsDN3o5Jh?q=gangnam&page=1&type=video&last_seen=Any">Yes!</a>
(Please, don’t tell me that it buffers… There’s <a href="https://fiatjaf.com/d5031e5b.html">NOTHING WRONG WITH THE DHT</a>! Eh!? Eh??) Anyways, <a href="https://n0.computer/blog/a-new-direction-for-iroh/">Iroh</a> is here to fix it all. 👋🙏
</figcaption>
</figure>Mathijs de BruinThis is the second post in a 2-post miniseries where we explain the challenges we faced scaling up, and how they were eventually overcome. In this second post we describe how we built a realistic benchmark. It details the problems which we faced scaling up to 73 nodes and how they were overcome by completely restructuring our indexes.1000 hits/s? Challenge accepted!2023-04-03T00:00:00-05:002023-04-03T00:00:00-05:00https://blog.ipfs-search.com/challenge-accepted<h1 id="introduction">Introduction</h1>
<p>In fall 2021 we started the ambitious work of seeing whether <a href="http://ipfs-search.com">ipfs-search.com</a> could truly handle web-scale traffic. Through the grapevine, we’d heard how a well known search engine might be interested in searching IPFS. Searching IPFS is what we do since 2016, so we said “challenge: accepted”.</p>
<p>The same grapevine told us that this search engine handles about 1000 requests per second. At the time, we were handling about 0.1 requests/second, so quite a difference. However, in our statistics we’re seeing early signs of exponential growth, in which case 4 orders of magnitude really doesn’t take that long.</p>
<p>Being passive on the internet means explosive growth will overwhelm you. Our success might well be the cause of our demise. Or worse; as with many start-ups we might feel forced to give up <a href="https://blog.ipfs-search.com/breaking-the-silent-consent/">our ideals</a> under market pressures, just to survive.</p>
<p>This is the first part in a two-post blog miniseries, where we describe how indeed we managed to surpass our ambitions of handling 1000 requests per second.</p>
<figure>
<img alt="Traffic growth over 2022." src="/assets/images/2023-04-03-challenge-accepted/api_requests.png" />
<figcaption>Traffic growth over 2022.</figcaption>
</figure>
<figure>
<img alt="Index growth over 2022." src="/assets/images/2023-04-03-challenge-accepted/documents_per_index.png" />
<figcaption>Index growth over 2022.</figcaption>
</figure>
<h1 id="its-elastic-right">It’s elastic, right?</h1>
<p>We were running Elastic (currently <a href="https://www.theregister.com/2021/04/13/aws_renames_elasticsearch_fork_opensearch/">OpenSearch</a>, as <a href="https://blog.opensource.org/the-sspl-is-not-an-open-source-license/">Elastic isn’t Open Source</a> anymore), a document store specifically designed to scale and handle gigantic datasets. After Google’s publication in the early 2000’s of <a href="https://en.wikipedia.org/wiki/MapReduce">MapReduce</a>, the smart folks behind Elasticsearch (amongst others) built a FOSS (Free and Open Source) search index with it. In theory, allowing scaling without a limit. However…</p>
<h2 id="theory-doesnt-work-in-practice">Theory doesn’t work in practice.</h2>
<p>Early benchmarks suggested that a single node was able to handle 10 queries per second. Which, again in theory, suggested that merely scaling out our cluster from 4 to 100 servers ought to do it. But alas, it wasn’t so easy.</p>
<p>As soon as we scaled our cluster from 4 up to 30 nodes, average response times shot up to over half a second! Mind you, these are averages — it implies that some of our users had to wait for several seconds for search results.</p>
<figure>
<img alt="Response times over 2021." src="/assets/images/2023-04-03-challenge-accepted/response_time.png" />
<figcaption>Response times over 2021.</figcaption>
</figure>
<h2 id="the-internet-is-impatient">The Internet is impatient!</h2>
<p>Unlike visitors of your local library, users have strong expectations when it comes to looking for information on the internet. Wait more than 200 ms and a website is experienced as slow. Wait more than 1 second and you’ll start interrupting the user’s flow. More than a few seconds and users will leave, never to return again. (<a href="https://ux.stackexchange.com/questions/100316/loading-time-and-user-expectations">Reference</a>) It doesn’t matter how many queries per second we can serve, it’ll be useless if we’re serving them too slowly!</p>
<h2 id="endless-fidgeting-with-knobs">Endless fidgeting with knobs</h2>
<p>Like any large and complex machine, Elastic/OpenSearch has a large number of configuration options which one can spend a lifetime tuning. Sadly enough, it seems that few experts in the field have bothered to share detailed knowledge. As soon as one leaves the ‘safe’ territory of the Proof of Concept, enter the domain of the Tech Consultant. Search being our core activity, this is potentially an endless sinkhole of funds, which we do not have in the first place!</p>
<figure>
<img alt="Control panel, knobs and dials." src="/assets/images/2023-04-03-challenge-accepted/Control-2-1200x766.jpg" />
<figcaption>Source: <a href="https://flashbak.com/the-control-panel-archive-the-tactile-beauty-of-buttons-meters-knobs-and-dials-406888/">https://flashbak.com/the-control-panel-archive-the-tactile-beauty-of-buttons-meters-knobs-and-dials-406888/</a></figcaption>
</figure>
<p>Rather than Outsourcing All The Things, we ended up becoming the consultants ourselves. Which is one of the reasons it took us over a year to learn how to overcome these obstacles, with the end result being that we now have all the knowledge in-house. (We did get some help, but more towards the practical side of the implementation.)</p>
<p>Over time, we tried:</p>
<ol>
<li><a href="https://github.com/ipfs-search/ipfs-search-deployment/blob/main/docs/architecture/sharding.pdf">Increasing the number of index shards</a> and…</li>
<li>… <a href="https://github.com/ipfs-search/ipfs-search-deployment/blob/main/docs/architecture/sharding%20reconsiderations%206-2-23.pdf">decreasing them again</a>.</li>
<li>Increasing the number of replicas and…</li>
<li>… decreasing them again.</li>
<li><a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/tune-for-indexing-speed.html#_unset_or_increase_the_refresh_interval">Tuning our refresh interval</a>.</li>
<li>Implementing <a href="https://github.com/ipfs-search/ipfs-search/pull/217">batching/bulk reads</a> and <a href="https://github.com/ipfs-search/ipfs-search/pull/201">writes</a> for our crawler.</li>
<li><a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/tune-for-indexing-speed.html#_indexing_buffer_size">Tuning our index buffer size</a>.</li>
<li><a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/tune-for-search-speed.html#_search_rounded_dates">Searching rounded dates</a>.</li>
<li>Upgrading from ElasticSearch to OpenSearch and…</li>
<li>Upgrading OpenSearch again.</li>
<li><a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html">Reindexing</a> All The Things, several times.</li>
<li>Ensuring persistent (keepalive) connections for search clients.</li>
<li><a href="https://www.outcoldman.com/en/archive/2017/07/13/elasticsearch-explaining-merge-settings/">Tuning max_merge_count</a> to prevent index throttling.</li>
<li>Reducing our crawling rate.</li>
<li>Enabling <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-cluster.html#shard-allocation-awareness">shard location awareness</a> and…</li>
<li>… disabling it again.</li>
<li>Tuning <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.17/search-shard-routing.html#search-concurrency-and-parallelism">max_concurrent_shard_requests</a> in search queries.</li>
<li>Enabling <code class="language-plaintext highlighter-rouge">_local</code> <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.17/search-shard-routing.html#shard-and-node-preference">shard preference</a> in queries and…</li>
<li>… disabling it again.</li>
<li>Setting per-shard search API <code class="language-plaintext highlighter-rouge">timeout</code>.</li>
<li>Set <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules-translog.html">translog</a> durability to async.</li>
<li><a href="https://www.exratione.com/2018/03/elasticsearch-adjusting-merge-settings-to-make-frequent-updates-less-painful/">Tuning <code class="language-plaintext highlighter-rouge">reclaim_deletes_weight</code></a>.</li>
</ol>
<h3 id="resources">Resources</h3>
<ul>
<li><a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/tune-for-indexing-speed.html#tune-for-indexing-speed">Tune for indexing speed</a> by Elasticsearch</li>
<li><a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/tune-for-search-speed.html">Tune for search speed</a> by Elasticsearch</li>
<li><a href="https://www.outcoldman.com/en/archive/2017/07/13/elasticsearch-explaining-merge-settings/">How to avoid index throttling, deep dive in segments merging</a> by Denis Gladkikh</li>
<li><a href="https://www.exratione.com/2018/03/elasticsearch-adjusting-merge-settings-to-make-frequent-updates-less-painful/">Adjusting Merge Settings to Make Frequent Updates Less Painful</a> by Reason</li>
<li><a href="https://www.alibabacloud.com/blog/alibaba-cloud-elasticsearch-performance-optimization_597092">Elasticsearch Performance Optimization</a> by Alibababa</li>
<li><a href="https://repost.aws/knowledge-center/opensearch-indexing-performance">How can I improve the indexing performance[…]?</a> by Amazon</li>
</ul>
<h2 id="reading-and-writing-but-not-at-the-same-time">Reading and writing, but not at the same time!</h2>
<p>It turns out there was not a single factor which could be clearly outlined as the ‘root cause’ of our issue, rather a number of factors was colluding. However, discovered there was resource contention between our crawler’s indexing and search queries. This is also why many of our measures focused on improving search through improving index performance. In the end, implementing asynchronous/bulk reads and writes significantly increased the stability of our cluster, reducing both the variance in response times as well as the average.</p>
<p>It did become clear though, as would be expected, that performing crawling in bulk and asynchronously was a major factor in getting our response times under control. And so in summer ‘22 it finally seemed we were ready to continue scaling, but…</p>
<figure>
<img alt="A glimpse of one of the monitoring dashboards which we developed along the process." src="/assets/images/2023-04-03-challenge-accepted/all_the_stats.png" />
<figcaption>A glimpse of one of the monitoring dashboards which we developed along the process.</figcaption>
</figure>
<h1 id="our-benchmark-havent-started-yet">Our benchmark haven’t started yet!</h1>
<p>Throughout the ‘minor’ delay and distraction of finally getting these darn response times under control, we went waaaay overboard creating extremely insightful monitoring dashboards. We implemented deep-reaching functionality in our crawler, of all components.</p>
<p>But we hadn’t yet managed to scale our cluster beyond 33 nodes! Nor develop or run our actual benchmark! Want to learn how we achieved this?</p>
<p>Continue reading <a href="https://blog.ipfs-search.com/searching-at-scale/">our second post</a>.</p>Mathijs de BruinIn fall 2021 we started the ambitious work of seeing whether [ipfs-search.com](http://ipfs-search.com) could truly handle web-scale traffic. Through the grapevine, we’d heard how a well known search engine might be interested in searching IPFS. Searching IPFS is what we do since 2016, so we said “challenge: accepted”.Decentralised search: from dream to reality2022-09-26T00:00:00-05:002022-09-26T00:00:00-05:00https://blog.ipfs-search.com/Decentralised%20search:%20from%20dream%20to%20reality<h1 id="decentralised-search-from-dream-to-reality">Decentralised search: from dream to reality</h1>
<p>At the beginning of May 2022, distributed web specialists from <a href="http://redpencil.io/">redpencil.io</a> and <a href="http://ipfs-search.com">ipfs-search.com</a> conducted an experiment to run a fully distributed search index at ipfs-search.com. The experiment was created and performed by Aad Versteden, together with Elena Poelman, and was based on the research by professor <a href="https://pietercolpaert.be/">Pieter Colpaert</a> of Ghent University.</p>
<p>In our short talk, Aad Versteden, also co-founder and CEO of redpencil.io, shares the general concept of decentralization, the design, and procedure of the experiment as well as his insights for the future of distributed search.</p>
<p><strong>ZM: How did <a href="http://redpencil.io/">redpencil.io</a> come to work with ipfs-search.com? What attracted you to them?</strong></p>
<p>AV: We showed interest in ipfs-search.com because discovery is a cornerstone of new web technologies. So, when we noticed that their services had gone down, we reached out to see if we could help. We work towards opening up the internet, and having a golden tool like ipfs-search.com disappear was not something we were going to ignore.</p>
<p>What’s more, we have chosen the distributed search route because running servers on this scale wouldn’t be financially feasible forever. As <a href="http://ipfs-search.com/">ipfs-search.com</a> grows, I think there will be a funding gap that we won’t be able to cover (the search is growing faster than <a href="http://redpencil.io/">redpencil.io</a>). The other team members there don’t have much experience with <a href="https://en.wikipedia.org/wiki/Linked_data">Linked Data</a> technologies. So it seems that there is scope for some breakthroughs.</p>
<p>As redpencil.io works on distributed knowledge with tangible applications, it made sense that we execute this experiment.</p>
<p><strong>Could you provide a little overview of the entire system in which the experiment was carried out?</strong></p>
<p>IPFS itself allows you to share resources through their network and lets you share what you have used with others, peer-2-peer. The concept is quite simple: when someone downloads a certain file and their neighbour also wants it, it could be shared directly instead of going through some centralised server. In the case of Netflix, for example, this means that if I am at my parents’ house with my brother, and we are separately watching the same Netflix series, that series will only be downloaded to our home network once.</p>
<table>
<thead>
<tr>
<th style="text-align: center"><img src="/assets/images/2022-09-26-distributed-search-interview/p2p.png" /></th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: center"><em>Michel Bakni, CC BY-SA 4.0 <a href="https://creativecommons.org/licenses/by-sa/4.0">https://creativecommons.org/licenses/by-sa/4.0</a>, via Wikimedia Common</em></td>
</tr>
</tbody>
</table>
<p>For search engines this is obviously a bit more complex, IPFS solves that for larger media files, but having a search index shared and used over the Internet is quite uncommon. There was research in the Linked Data space on how we can build resources that are shared and discoverable. If we consider the search index as a big folder of files hosted on IPFS, we find that we can reuse some of the technologies—mainly research by professor Pieter Colpaert.</p>
<p>What they have done is to say—if we are going to have a dataset, and we want to get information out of it, we shouldn’t be running a very heavy server to do that because then we are the ones who have to pay for that server. It’s better for the end users to have a slightly higher cost per query and for us, the providers, to have a vastly lower cost. The cheapest way to do something like that is basically to say: look, here’s the data in an index, go and figure out how to reuse it.</p>
<p>Sharing the index as a whole would mean people downloading gigabytes of data to answer a query. Nobody wants to do that, and it is not feasible.</p>
<p>So, prof. Colpaert found a way to split this data to retrieve only what is needed to perform a query. Purely by using Linked Data technologies. There is a solution for search engines, prefix search, and also for full-text search, but we haven’t tried full-text search.</p>
<p><strong>What have you tried?</strong></p>
<p>We implemented prefix search. It means that we took the full 2019 and 2020 datasets from <a href="http://ipfs-search.com/">ipfs-search.com</a>, and created a split version of it. We had all the titles and looked at what letter they start with. The way it works is if someone searches for a title that starts with a ‘T’, they will be redirected to a page of the index with that letter or combination of letters. It narrows down the results so that for each letter searched—only one page is retrieved. These pages are small parts of the search index, so if my spouse also searched for the same letter(s), I would automatically provide her with my part of the index. It would not go through anyone else, it would remain on the local network.</p>
<table>
<thead>
<tr>
<th style="text-align: center"><img src="/assets/images/2022-09-26-distributed-search-interview/3.png" /></th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: center"><em>Frontend view. Courtesy of <a href="http://redpencil.io/">redpencil.io</a></em></td>
</tr>
</tbody>
</table>
<p>Prefix search allows you to search for the beginning of the page title only. It breaks down the search query into letters and creates some sort of container for each letter’s results. It keeps narrowing down the search results until you get all records from the index that starts, let’s say, with “The sta”. This is great progress, but it is similar to an index of the library more than to the search engines we are familiar with nowadays.</p>
<p><strong>So what exactly did this experiment involve?</strong></p>
<p>Our experiment consisted of taking the <a href="http://ipfs-search.com/">ipfs-search.com</a> database, titles, and some identifiers so that we knew where to find these resources, partitioning it to enable this type of search using known technologies, publishing the full dataset on IPFS, and then building a frontend hosted on IPFS. If someone wanted a full search index, they could <a href="https://docs.ipfs.tech/how-to/pin-files/">pin</a> that folder to have it locally available. This is useful in cases where someone wants to host it to make it easily accessible.</p>
<p>We have some benchmarks that are great user experience, but for some others, it was more a proof of concept than a usable tool. For example, when we hosted it on one node, it was quite slow at times. With 3 nodes, the content was already faster to access: it took a few seconds to get to the first page, and then it would go on quickly. With 4 nodes, we needed a second to download the first page, and subsequent pages took about 250 milliseconds. Of course, for already searched keywords, the results appear faster, so you can see them as they are discovered. The more people use the index, the faster it becomes.</p>
<p><strong>What downsides did this approach have?</strong></p>
<p>Well, it’s a fully distributed search index in the sense that the index itself is shared and it’s a bit strange to even be possible and a bit strange that it actually works haha.</p>
<p>However, the search index is built centrally by one entity that says: this is the index, you should trust us. The same way as it is with <a href="http://ipfs-search.com">ipfs-search.com</a>. Suboptimal but this is the reality for now.<br />
The other downside is updating the index – every month it will be full of new pages, hence the data you cached and shared with others will carry no value anymore. So that’s a bit problematic. But improvements are very feasible and possible.</p>
<p>Another one is the fact that we didn’t build a full-text search; on ipfs-search.com, you very often search for a topic rather than a title you already know. A full-text index would be more useful for end users.</p>
<p><strong>If we imagine a fully distributed search engine, what would it look like in practice?</strong></p>
<p>When you search a query, you have a need for certain parts of a large database. What happens now is that <a href="http://elastic.co">Elasticsearch</a>, which is used server-side at ipfs-search.com, gets a set of results, and to compute that, it will need to use parts of its index. It will combine them and come up with 50 results that might be of interest to the user.</p>
<p>In the semantic web, where the idea that everything should be decentralised and discoverable is prevalent<em>,</em> the approach would be different. It would be to take the search index and cut it into a million pieces that the user can retrieve.</p>
<p>Imagine you view an image via ipfs-search.com. This means that the image will be in your cache for a while and then forgotten. But in case someone else asks for the same image earlier, you can offer it to them.</p>
<p>The same will happen with all tiny pieces of the index you downloaded, as long as they are cached, you can share them with other peers, and effectively host them.</p>
<p>In case ipfs-search.com ceases to exist, the index remains alive, without pinning it anywhere, and will still be available through peers that are using it. If enough people have its bits and pieces, with some luck we will still have a full index.</p>
<p>It’s worth mentioning that even if some pieces are missing, it means that they were of no interest to the user, no one was looking for them.</p>
<table>
<thead>
<tr>
<th style="text-align: center"><img src="/assets/images/2022-09-26-distributed-search-interview/centralised.png" /></th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: center"><em>CC BY-SA 3.0 <a href="http://creativecommons.org/licenses/by-sa/3.0/">http://creativecommons.org/licenses/by-sa/3.0/</a>, via Wikimedia Commons</em></td>
</tr>
</tbody>
</table>
<p>It is also not a stretch to imagine that a user will trust and choose certain indexes. For example, a user decides to trust ipfs-search.com, and also, their university’s search engine, and wants to combine the information gathered by these indexes. It is possible to create a space, where people can search through entities they want to. If that’s possible, it’s also possible to have a distributively constructed search. And it is not only about trust because sometimes you want to look for something via a source you don’t trust so much.</p>
<p>When we did the experiment, we found it exceptional that we could have something working without a huge central database that provides search, that can be commoditised and done by people… Go back 15 years, and it would be a threat to some major industries.</p>
<p><strong>So, there is hope?</strong></p>
<p>There is hope. A lot of it. If people want their communities to find stuff, and they don’t want to contaminate other communities with it, then we can build a distributed search. There will be a lot of research on human behaviour to be done and experiments like “does it explode today or not?”.</p>
<p>But I think it’s feasible and the technologies we have today are a good start. It’s extremely promising. But also we need to be very realistic – this is not something that is going to replace the main search engine within five years or so, because there will be a lack of functionality. If there is full-on research into it, then yes, totally. But this is not what is happening right now.</p>
<p><strong>Do you and your team have any plans regarding running an expanded version of the current experiment?</strong></p>
<p>If possible, we should go towards larger data sets. We notice exponential growth in the search index, and we also noticed that the way how we now build the search index can’t keep up with the growing database, it gradually becomes slower. It was the first experiment and we know how to counter the issues. Great results for a proof of concept.</p>
<p>We’ll have to see what the performance impact is of running across nodes at some point and what the impact is for full-text search, but we are very much in the game.</p>
<h2 id="further-readingwatching">Further reading/watching:</h2>
<ul>
<li><a href="https://github.com/redpencilio/ldes-publisher-service">https://github.com/redpencilio/ldes-publisher-service</a></li>
<li><a href="https://github.com/redpencilio/ldes-prefix-autocomplete">https://github.com/redpencilio/ldes-prefix-autocomplete</a></li>
<li><a href="https://github.com/redpencilio/fragmentation-producer-service">https://github.com/redpencilio/fragmentation-producer-service</a></li>
<li><a href="https://www.ted.com/talks/pieter_colpaert_open_data_to_create_power_for_the_many_not_the_few">Ted Talk with Pieter Colpaert</a></li>
</ul>
<h1 id="how-to-run-the-application-at-home">How to run the application at home:</h1>
<h2 id="preparing-your-ipfs-daemon">Preparing your IPFS daemon</h2>
<p>The ipfs daemon can be configured. The way the application is currently hosted, it expects <code class="language-plaintext highlighter-rouge">Access-Control-Allow-Origin</code> or be set to <code class="language-plaintext highlighter-rouge">*</code>. This means any website can request any resource over IPFS. This shouldn’t be able to cause any harm when on public IPFS.</p>
<h3 id="easy-route">Easy route</h3>
<p><code class="language-plaintext highlighter-rouge">ipfs config "Access-Control-Allow-Origin" "[*]"</code></p>
<h3 id="complete-route">Complete route</h3>
<p>All settings can be configured through</p>
<p><code class="language-plaintext highlighter-rouge">ipfs config edit</code></p>
<p>This guide assumes you have the IPFS companion up and running with your own gateway and that your gateway has the Access-Control-Allow-Origin set to <code class="language-plaintext highlighter-rouge">["*"]</code> as in:</p>
<p><code class="language-plaintext highlighter-rouge">{
"API": { "Access-Control-Allow-Origin": ["*"] }
}</code></p>
<h2 id="opening-up-the-frontend">Opening up the frontend</h2>
<p>We assume you’re running the IPFS companion which redirects calls to <code class="language-plaintext highlighter-rouge">ipfs://</code> to your local daemon.</p>
<p>Visit the frontend at ipfs://ipfs/QmXiKm8Y37YyNWsX3bMNpMEHuoUCKWkWvPVUFGP2Ex9kq6</p>
<h2 id="entering-strange-information">Entering strange information</h2>
<p>The frontend allows you to browse different indexes. We’ve made the starting point of a full-text search index available at <a href="https://gateway.ipfs.io/ipfs/QmbJT8MRZnyv8gYQmcmUk8FYdgqJFwrn6634CCtxiPd3xr/1">https://gateway.ipfs.io/ipfs/QmbJT8MRZnyv8gYQmcmUk8FYdgqJFwrn6634CCtxiPd3xr/1</a></p>
<p>under the ttl format.</p>
<p>Enter the aforementioned URL in the <em>first text input.</em></p>
<p>Pick .ttl as a format and click <code class="language-plaintext highlighter-rouge">SET DATASOURCE</code>.</p>
<p><img src="/assets/images/2022-09-26-distributed-search-interview/1.png" /></p>
<p>You can verify the first page was fetched by opening your network tab. <em>Notice 1.ttl has been fetched.</em></p>
<h2 id="searching">Searching</h2>
<p>You can now enter any search query. Results are fetched live using a prefix search index.</p>
<p>As you type results, the pages for each letter of the query are fetched. Sometimes the network doesn’t find its way and it takes a while to find the specific page.</p>
<p><img src="/assets/images/2022-09-26-distributed-search-interview/2.png" /></p>
<p>And sometimes it goes very fast:</p>
<p><img src="/assets/images/2022-09-26-distributed-search-interview/3.png" /></p>Zuzanna MajerAt the beginning of May 2022, distributed web specialists from [redpencil.io](http://redpencil.io/) and [ipfs-search.com](http://ipfs-search.com) conducted an experiment to run a fully distributed search index at ipfs-search.com.Anatomy of a search engine2022-09-11T00:00:00-05:002022-09-11T00:00:00-05:00https://blog.ipfs-search.com/Anatomy-of-a-search-engine<h1 id="anatomy-of-a-search-engine">Anatomy of a search engine</h1>
<p>In previous posts, we’ve covered the development of <a href="https://blog.ipfs-search.com/NSFW-f70ee/">frontend filters</a>, described progress on <a href="https://blog.ipfs-search.com/scaling-up-the-search/">scaling up the cluster architecture</a>, and glanced at the <a href="https://blog.ipfs-search.com/breaking-the-silent-consent/">importance of web security</a>.</p>
<p>Now it is time to dive a little deeper into what ipfs-search.com, and basically any modern search engine, consists of.<br />
As this is a very complex topic, we will take the liberty here of viewing just a few selected elements.</p>
<div align="center">
<img src="/assets/images/Anatomy_of_a_search_engine/documentslastmonth.png" /></div>
<hr />
<p>Our latest statistics show that our index is growing rapidly. We store 20 TB of searchable data. Currently, every day, half a million documents are added to the index.</p>
<p>Let’s take a look at how is it done. Here we have the elements that are responsible for catching this data, classifying it, and giving them correct labels.</p>
<h3 id="a-network-sniffer">A network sniffer</h3>
<p>If you go through <a href="https://ipfs-search.readthedocs.io/en/latest/architecture.html">ipfs-search.com docs</a>, you can read in the documentation that our search “…sniffs the DHT gossip and indexes file and directory hashes”.</p>
<p>Sounds cool, but what does that even mean?</p>
<table>
<thead>
<tr>
<th style="text-align: center"><img src="/assets/images/Anatomy_of_a_search_engine/Dogs_sniffing_each_other.jpg" /></th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: center"><em>Jurriaan Schulman, CC BY-SA 3.0 <a href="http://creativecommons.org/licenses/by-sa/3.0/">http://creativecommons.org/licenses/by-sa/3.0/</a>, via Wikimedia Commons</em></td>
</tr>
</tbody>
</table>
<p>When we send information over a computer network, it is broken down into smaller units. They are the smallest units of network communication, called data packets. The sender’s node (which is just a device connected to a network) breaks down each piece of information into these smallest units, and after completing their journey to the receiver’s node, they are reassembled into their originals.</p>
<table>
<thead>
<tr>
<th style="text-align: center"><img src="/assets/images/Anatomy_of_a_search_engine/Network_packet.jpg" /></th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: center"><em><a href="https://commons.wikimedia.org/wiki/File:Network_packet.jpg">https://commons.wikimedia.org/wiki/File:Network_packet.jpg</a>, via Wikimedia Commons</em></td>
</tr>
</tbody>
</table>
<p>Data packets are commonly monitored by sysadmins for security reasons, to search for anomalies in traffic, and perform maintenance.</p>
<p>Intercepting data packets on a computer network is called packet sniffing, and it’s a term that is normally used in information security and network diagnostics. We recognize two ways of using it, legal and not so. It’s often how our <a href="https://en.wikipedia.org/wiki/XKeyscore">governments listen in on our private communication</a> and in the past, was commonly used by hackers for identity theft — stealing credit cards, passwords, etc. (Nowadays, most communication is encrypted, but creepy organisations like the NSA <a href="https://www.wired.com/2012/03/ff-nsadatacenter/">store all of your data and are likely able to break even modern strong encryption</a>.)</p>
<p>The sniffing process looks similar to wiring a phone or eavesdropping behind the door, although it requires way more than only gathering data.</p>
<p>A sniffer itself is a piece of software (like, for example, <a href="https://www.wireshark.org/">Wireshark</a>, which provides GUI and some helpful analytics tools) that you connect to a computer network to see the traffic.</p>
<table>
<thead>
<tr>
<th style="text-align: center"><img src="/assets/images/Anatomy_of_a_search_engine/Wireshark_Example_Decode.png" /></th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: center"><em>Wireshark, CC BY-SA 4.0 <a href="https://creativecommons.org/licenses/by-sa/4.0">https://creativecommons.org/licenses/by-sa/4.0</a>, via Wikimedia Commons</em></td>
</tr>
</tbody>
</table>
<h3 id="ipfs-searchcom-sniffer">ipfs-search.com sniffer</h3>
<p><a href="https://github.com/ipfs-search/ipfs-search/tree/master/components/sniffer">Our sniffer</a> does not commit any crimes though. It’s based on the existing <a href="https://github.com/libp2p/hydra-booster">Hydra-Booster,</a> “A new type of DHT (Distributed Hash Tables) node designed to accelerate the Content Resolution & Content Providing on the IPFS Network. A (cute) Hydra with one belly full of records and many heads (Peer IDs) to tell other nodes about them, charged with rocket boosters to transport other nodes to their destination faster.”</p>
<table>
<thead>
<tr>
<th style="text-align: center"><img src="/assets/images/Anatomy_of_a_search_engine/hydra_booster.png" /></th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: center"><em>Hydra-booster</em></td>
</tr>
</tbody>
</table>
<p>To make it more useful for our purposes, we created a <a href="https://pkg.go.dev/github.com/ipfs-search/ipfs-search@v0.0.0-20220720103450-c3d9687780aa/components/sniffer">‘middleware’/proxy</a> between the part in IPFS/libp2p that stores what hosts have, so that every time it learns about something new, it gets passed to our crawler infrastructure.</p>
<p>Our sniffer is currently run on a single node, where we do deduplication of sniffed content. We are upgrading our architecture to allow for distributed sniffing of new content from IPFS’s DHT.</p>
<blockquote>
<p>📢 ipfs-search.com sniffer currently uses 12 heads to process about 3000 hashes per second.</p>
</blockquote>
<h3 id="gossip">Gossip</h3>
<p>Then we just need gossip.</p>
<table>
<thead>
<tr>
<th style="text-align: center"><img src="/assets/images/Anatomy_of_a_search_engine/gossip1.jpg" /></th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: center"><em>CC-BY-NC-SA 4.0 via <a href="http://www.slenquirer.com/2014/04/gossip-in-sl-aint-nobody-got-time-for.html">SL Enquirer</a></em></td>
</tr>
</tbody>
</table>
<p>Exactly the same way when people go to the café to exchange important or less important information, in a peer-to-peer network (like Libp2p/IPFS, BitTorrent, or other content-addressed storage systems) nodes talk to other nodes about the content they have.</p>
<table>
<thead>
<tr>
<th style="text-align: center"><img src="/assets/images/Anatomy_of_a_search_engine/BitTorrent_network.svg" /></th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: center"><em>Scott Martin, CC BY-SA 3.0 <a href="https://creativecommons.org/licenses/by-sa/3.0">https://creativecommons.org/licenses/by-sa/3.0</a>, via Wikimedia Commons</em></td>
</tr>
</tbody>
</table>
<p>They have rather simple conversations going on, like “Where is this file? Have you seen it?”, “Which node has it?”, “It was here, but now it’s there.” etc.</p>
<blockquote>
<p>📢 So how does ipfs-search.com do content discovery? How do we know what’s on IPFS?</p>
</blockquote>
<p>For the network, we’re just a bunch of nodes, we listen to other nodes announcing what’s available. When we hear the message saying “I have this file, you can download it from me” a small signal passes through our network, and our crawler (the infrastructure that extracts metadata) gets the file and indexes it.</p>
<p>We store them in our database which lives on the cluster consisting of several servers which each index and search about 2 TB. So on the one side, we have crawlers that capture, index, and extract metadata whenever the sniffer finds new content, and on the client side, there is <a href="https://ipfs-search.com">ipfs-search.com</a>, our beautiful frontend. When a user searches for something, they talk to our database, and this is where the result of their query comes from.</p>
<h3 id="a-crawler">A crawler</h3>
<table>
<thead>
<tr>
<th style="text-align: center"><img src="/assets/images/Anatomy_of_a_search_engine/3458011826_ec2838a13c_o.jpg" /></th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: center"><em>CC BY-NC-SA 2.0 by <a href="https://www.flickr.com/photos/torek/3458011826">Héctor García</a></em></td>
</tr>
</tbody>
</table>
<p>A typical search engine also works with web crawlers. A crawler, or sometimes web spider, or, surprisingly, a spiderbot, is a bot, another piece of software, that visits webpages and indexes content that is uploaded by the users. It is also necessary to keep this content up to date and can be helpful with validating hyperlinks or HTML code.</p>
<table>
<thead>
<tr>
<th style="text-align: center"><img src="/assets/images/Anatomy_of_a_search_engine/ipfs-search-arch-inv.png" /></th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: center"><em>Sketch of the ipfs-search.com architecture</em></td>
</tr>
</tbody>
</table>
<p>The ipfs-search.com crawler is also the component that orchestrates the process of extracting metadata from all data that is flowing through our network.</p>
<p>For this job, we use <a href="https://tika.apache.org/">Apache’s Tika</a>, for which we developed the highly efficient streaming <a href="https://github.com/ipfs-search/tika-extractor">tika-extractor</a>, that gets a blob of bits and bytes thrown at its server by the crawler and puts a label: This is a music file, that is a text file, these are an author and a title… We made a special component that asynchronously requests data over our IPFS node, which makes this process more efficient.</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="nl">"metadata"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"xmpDM:genre"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="s2">"Soundtrack"</span><span class="w">
</span><span class="p">],</span><span class="w">
</span><span class="nl">"xmpDM:composer"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="s2">"Nobuo Uematsu"</span><span class="w">
</span><span class="p">],</span><span class="w">
</span><span class="nl">"X-Parsed-By"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="s2">"org.apache.tika.parser.DefaultParser"</span><span class="p">,</span><span class="w">
</span><span class="s2">"org.apache.tika.parser.mp3.Mp3Parser"</span><span class="w">
</span><span class="p">],</span><span class="w">
</span><span class="nl">"creator"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="s2">""</span><span class="w">
</span><span class="p">],</span><span class="w">
</span><span class="nl">"xmpDM:album"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="s2">"</span><span class="se">\"</span><span class="s2">Final Fantasy IX</span><span class="se">\"</span><span class="s2"> Original Soundtrack, Disk 4"</span><span class="w">
</span><span class="p">],</span><span class="w">
</span><span class="nl">"xmpDM:trackNumber"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="s2">"24"</span><span class="w">
</span><span class="p">],</span><span class="w">
</span><span class="nl">"xmpDM:releaseDate"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="s2">"2000"</span><span class="w">
</span><span class="p">],</span><span class="w">
</span><span class="nl">"meta:author"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="s2">""</span><span class="w">
</span><span class="p">],</span><span class="w">
</span><span class="nl">"xmpDM:artist"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="s2">""</span><span class="w">
</span><span class="p">],</span><span class="w">
</span><span class="nl">"dc:creator"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="s2">""</span><span class="w">
</span><span class="p">],</span><span class="w">
</span><span class="nl">"xmpDM:audioCompressor"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="s2">"MP3"</span><span class="w">
</span><span class="p">],</span><span class="w">
</span><span class="nl">"resourceName"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="s2">"24-Coca Cola TV CM 1.mp3"</span><span class="w">
</span><span class="p">],</span><span class="w">
</span><span class="nl">"title"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="s2">"Coca Cola TV CM 1"</span><span class="w">
</span><span class="p">],</span><span class="w">
</span><span class="nl">"xmpDM:audioChannelType"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="s2">"Stereo"</span><span class="w">
</span><span class="p">],</span><span class="w">
</span><span class="nl">"version"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="s2">"MPEG 3 Layer III Version 1"</span><span class="w">
</span><span class="p">],</span><span class="w">
</span><span class="nl">"xmpDM:logComment"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="s2">"eng - </span><span class="se">\n</span><span class="s2">http://www.ffdream.com"</span><span class="w">
</span><span class="p">],</span><span class="w">
</span><span class="nl">"xmpDM:audioSampleRate"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="s2">"44100"</span><span class="w">
</span><span class="p">],</span><span class="w">
</span><span class="nl">"channels"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="s2">"2"</span><span class="w">
</span><span class="p">],</span><span class="w">
</span><span class="nl">"dc:title"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="s2">"Coca Cola TV CM 1"</span><span class="w">
</span><span class="p">],</span><span class="w">
</span><span class="nl">"Author"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="s2">""</span><span class="w">
</span><span class="p">],</span><span class="w">
</span><span class="nl">"xmpDM:duration"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="s2">"20218.76953125"</span><span class="w">
</span><span class="p">],</span><span class="w">
</span><span class="nl">"Content-Type"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="s2">"audio/mpeg"</span><span class="w">
</span><span class="p">],</span><span class="w">
</span><span class="nl">"samplerate"</span><span class="p">:</span><span class="w"> </span><span class="s2">"-</span><span class="se">\"</span><span class="s2">44100</span><span class="se">\"</span><span class="s2">"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"version"</span><span class="p">:</span><span class="w"> </span><span class="mi">2</span><span class="p">,</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"file"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<h3 id="a-bitswap-protocol">A bitswap protocol</h3>
<p>It is worth mentioning, that IPFS is built on the protocol called <a href="https://docs.ipfs.tech/concepts/bitswap/">bitswap</a> where basically nodes trade data, exchanging a <em>want-have</em> request. If you want to download something, the way to get it is to have something that somebody else wants. This is how the network balances itself.</p>
<h3 id="summary">Summary</h3>
<p>So basically, what ipfs-search.com does is: while nodes (all the computers that are connected to IPFS) talk to each other about available resources, the sniffer (another node), listens to this communication, and extracts hashes. When something is interesting, the crawler extracts data from the hashes and indexes them.</p>
<p>Of course, there is more to it. There are other processes under the hood, such as queuing, which is done using RabbitMQ, or our <a href="https://github.com/ipfs-search/ipfs-search-api/">search API microservice</a>. We refer those interested to our <a href="https://ipfs-search.readthedocs.io/en/latest/">documentation</a>.</p>
<h3 id="taking-it-further">Taking it further</h3>
<p>In April <a href="http://protocol.ai">Protocol Labs</a> released the first production of the <a href="https://filecoin.io/blog/posts/introducing-the-network-indexer/">Network Indexer</a> which makes searching (by CID or multihash) content-addressable data networks like IPFS and Filecoin possible. This is a decisive step towards a goal that also is in our line of work: easier and more accessible fetching of data across the IPFS network.</p>
<p>We might be looking at the option of combining these two indexing technologies. The result could be exciting.</p>
<p>Also, we’ll be moving to a different queuing system where we can have multiple sniffers and/or have them integrated with our IPFS nodes.</p>
<p><strong>Resources:</strong></p>
<ul>
<li>
<p><a href="https://github.com/ipfs-search/ipfs-search/tree/master/components/sniffer">https://github.com/ipfs-search/ipfs-search/tree/master/components/sniffer</a></p>
</li>
<li>
<p><a href="https://github.com/ipfs-search/tika-extractor">https://github.com/ipfs-search/tika-extractor</a></p>
</li>
<li>
<p><a href="https://pkg.go.dev/github.com/ipfs-search/ipfs-search@v0.0.0-20220404092707-198591df419c/components/sniffer">https://pkg.go.dev/github.com/ipfs-search/ipfs-search@v0.0.0-20220404092707-198591df419c/components/sniffer</a></p>
</li>
<li>
<p><a href="https://github.com/libp2p/hydra-booster/commit/d8438c7b58d7f3639c22252e97873c42617cf389">https://github.com/libp2p/hydra-booster/commit/d8438c7b58d7f3639c22252e97873c42617cf389</a></p>
</li>
</ul>Zuzanna MajerOur latest statistics show that our index is growing rapidly. We store 20 TB of searchable data. Currently, every day, half a million documents are added to the index.Scaling up the search2022-07-01T00:00:00-05:002022-07-01T00:00:00-05:00https://blog.ipfs-search.com/scaling-up-the-search<p>As some of you know, we are supported by <a href="https://nlnet.nl/project/IPFS-search/">NLNet</a> through the EU’s <a href="https://www.ngi.eu/">Next Generation Internet (NGI0)</a> programme, which stimulates network research and development of the free Internet, to do the architecture for scaling up our infrastructure. We are additionally supported by the <a href="https://fil.org/">Filecoin Foundation</a>, who support the growth of the distributed web, through a <a href="https://github.com/ipfs-search/devgrants/blob/96d07f4662c10f3936163dacbf13bfc5c23b8cc6/open-grant-proposals/ipfs-search-scale-out.md">devgrant</a>, whichs helps us to actually implement the scale-out.</p>
<p>We successfully followed our plan to move step-by-step from one server to a 5 node cluster setup, then to 15 servers, and now we are scaling up through 30, 50 and up to 100 nodes. This puts us on the path to 1000 hits per second; a thousand users every second searching something. We are now in the middle of the way, running on 30 servers. The current experiment is for us to <em>learn how</em> to scale our infrastructure up, until 100 nodes.</p>
<p>Right now, we have indexing capacity of 20 TB, and we are planning to have 100 TB by the end of our scale-out experiment. It is a real challenge as a typical computer stores around 1 TB and copying this 1 TB from one computer to another can take hours.</p>
<p><img src="/assets/images/3servers.png" /></p>
<hr />
<p><img src="/assets/images/5servers.png" /></p>
<h2 id="but-let-us-walk-you-through-what-have-been-going-on-in-our-headquarters-recently"><strong>But let us walk you through what have been going on in our headquarters recently</strong></h2>
<p>One of our ways to limit costs is to use physical servers instead of, very popular, cloud servers. This choice is also recommended by Elasticsearch, which we use. After a careful research, we have chosen Hetzner hosting, a German company that provides climate neutral servers, which was also important for us. Why exactly we decided to use bare metal servers? We like to keep an eye on what’s going on. We are able to track temperature and delays on individual discs, we know about every hardware failure, every unusual behaviour pattern and if we were using virtual server we wouldn’t know all these things. Also, the costs are about a factor 10 lower, because we use a lot of data, memory, storage, CPU and I/O.</p>
<p>In the beginning we have been indexing on one server, the most powerful server at Hetzner’s and of course at one point it ran full. We had to shut down the indexing, because we weren’t able to take new files. All this was caused by the fact that in the previous year we made some changes to the crawler (the part that extracts data from the hashes and indexes them) that made it about 100 times faster. So suddenly, instead of indexing 0.1 document per second, we were indexing about 10 documents per second. The consequence was obvious — scaling up the hosting.</p>
<h3 id="-we-werent-expecting-a-totally-smooth-transition-as-we-know-that-designing-a-perfect-cluster-is-almost-impossible-at-the-beginning">🛠 We weren’t expecting a totally smooth transition, as we know that designing a perfect cluster is almost impossible at the beginning.</h3>
<p>So, when we went up to 2 servers, and there were no problems, it was a great surprise. Our deployments are automated, we are using Ansible. This allowed us in the past to change a hosting company in about two days. It is a reasonable solution to deal with multiple servers. Instead of executing a gazillion commands for every server manually, and checking the results, Ansible does this for us. But the architecture, what server does what, and telling that in the correct way to Ansible, was the challenge.</p>
<div align="center">
<img src="/assets/images/graf1.png" />
</div>
<h2 id="redundancy">Redundancy</h2>
<p>Later we moved to 3 servers, and we had reached the point where when something breaks in one of them, the page is still up. If you design a larger server architecture, there will always be, depending on the size of your system, some number of servers that perform badly. They are guaranteed to crash at one point, and by expecting it, there will be no degradation of the service as a result. However, to be safe while this is happening, we needed to prepare a fault-tolerant cluster. It means, among other things, distributing sliced parts of data (called shards) between multiple nodes. Then, creating a copy of every shard and allocating it to a different node in a way that no original and its copy live on the same node. The replica shard is always up-to-date with the original. That make sure that even if some servers are down, all the data is available.</p>
<p>These shards, logical and physical divisions of an index, need to be tuned really carefully to the size of the server and size of our constantly growing data.</p>
<p>Although it comes with some disadvantages, horizontal partitioning, by reducing index size, greatly improves search performance.</p>
<h2 id="coordination-through-dedicated-master-nodes">Coordination through dedicated master nodes</h2>
<p>We also introduced the difference between data and master nodes. Master nodes take care of allocating chosen shards of our data on chosen servers, and making sure the servers know about each other. They are also maintaining information including shards’ localization (which node are they on), index mapping, and performing healthchecks. We have to adjust numbers of data nodes to the growing architecture in order to maintain cluster stability, but the amount of master nodes always stays at 3.</p>
<h2 id="data-replication">Data replication</h2>
<p>Last but not least, we were working on data replication. IPFS Search by definition is an entry to a lot of data, which must not be lost. We set our replication factor to 2 which means that we keep 3 copies of data in our cluster, 1 primary and 2 replicas. In other words, even if a primary is lost, its replica can be made a primary until the recovery.</p>
<p>In addition to this, we make daily snapshots of our index, so that even if we accidentally delete all our data (e.g. human error or end of the World…) we keep a backup.</p>
<p>So we came a long way from 1 document to 500 documents being indexed or updated at the same time, and we’re still improving and optimizing various part of this system. The challenge here was (and still is) finding a golden way to tune the shards, and keep our cluster healthy and balanced.</p>
<div align="center">
<img src="/assets/images/nodes.png" />
</div>Zuzanna MajerFor the past few months, we have been working on the search architecture to take IPFS Search from beta to web-scale productionBreaking the silent consent - closer to the free Internet, an interview with the founder of IPFS Search2022-06-03T00:00:00-05:002022-06-03T00:00:00-05:00https://blog.ipfs-search.com/breaking-the-silent-consent<h1 id="breaking-the-silent-consent---closer-to-the-free-internet-an-interview-with-the-founder-of-ipfs-search">Breaking the silent consent - closer to the free Internet, an interview with the founder of IPFS Search</h1>
<p>Online privacy and security are too rarely questioned by ordinary users. Taking them for granted comes from the fact that most people believe they have control over the information they share. Most of us live in silent consent to something that, in a non-virtual society, we would never give permission for. Not everyone has the time to use alternative tools, search engines, browsers, plug-ins, and a range of security features. Learning new things takes time, and with that, as with everything else, the primacy of convenience wins out.</p>
<p><img src="/assets/images/2021-06-03-breaking-the-silent-consent/Artboard_1.svg" /></p>
<p>Still, there is a safer future for the Internet, and it’s us, users, who should fight to bring this future into today.</p>
<p>Our guest is Mathijs de Bruin, founder, and inventor of <a href="http://ipfs-search.com/">IPFS Search</a>, a search engine indexing the open-source Interplanetary File System which describes itself as “a peer-to-peer hypermedia protocol designed to preserve and grow humanity’s knowledge by making the web upgradeable, resilient, and more open.”</p>
<p><img src="/assets/images/2021-06-03-breaking-the-silent-consent/Untitled.png" /></p>
<p><strong>ZM:</strong> <strong>Maybe before we really start this talk, tell us how this description of the project you contribute to, resonates with you?</strong></p>
<p><strong>MdB:</strong> I think what <a href="https://twitter.com/juanbenet">Juan Benet</a> is striving for is something that we have been looking for in the hacker movement for a long time.</p>
<p>So, as hacker technologists, we are the people that maintain the Internet infrastructure. For most users, the Internet is something that is just “there.” Like a black box, you put a few cables together and the magic happens. But for us, it is different, we know what’s going on inside this box.</p>
<p>Let’s remember that initially the Internet was set up as a research protocol, among research institutions, mostly by the US government. It was created to be resilient against outsiders’ attacks, specifically nuclear attacks. But it wasn’t set up to be resilient to censorship or cyberattacks.</p>
<table>
<thead>
<tr>
<th style="text-align: center"><img src="../assets/images/2021-06-03-breaking-the-silent-consent/Internet_map_1024.jpg" alt="Internet Map" title="image" /></th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: center"><em>Internet backbone as of January 15, 2005. <a href="https://commons.wikimedia.org/w/index.php?curid=1538544">CC BY 2.5</a></em></td>
</tr>
</tbody>
</table>
<p>The way I see it is that the only reason we have free internet right now is because a lot of people that are maintaining core infrastructure have very strong morals and principles. It is not accidental that the Internet used to be an open, free protocol. Now we have mobile providers who offer <em>free</em> Internet for Facebook, YouTube, and Spotify, but not in general. And at the same time, we see that Facebooks and YouTubes of this world are applying <a href="https://www.bmj.com/content/375/bmj.n2635/rr-80">various</a> <a href="https://theintercept.com/2020/10/15/facebook-and-twitter-cross-a-line-far-more-dangerous-than-what-they-censor/">kinds</a> of <a href="https://www.eff.org/deeplinks/2020/03/ninth-circuit-private-social-media-platforms-are-not-bound-first-amendment">censorship</a> to a frightening degree. We are talking about free, democratic societies. It’s a complete erosion of civil liberties that… we didn’t really have until the advent of cyberspace, and now we are already looking to lose them. We’ve been worried about that in the hacker community for a long time.</p>
<table>
<thead>
<tr>
<th style="text-align: center"><img src="/assets/images/2021-06-03-breaking-the-silent-consent/7170226404_3a9a96976d_o.jpg" /></th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: center"><em>CC BY-NC-SA 2.0 by <a href="https://www.flickr.com/photos/florianhauschild/7170226404/">Florian Hauschild</a></em></td>
</tr>
</tbody>
</table>
<p>In countries that are not pretending to be free, the Internet has been cut off or censored shamelessly. For example, <a href="https://blog.ipfs.io/24-uncensorable-wikipedia/">Turkey blocked Turkish Wikipedia</a> because of an article about state-sponsored terrorism. Spain has been blocking access to information about the <a href="https://www.trustnodes.com/2017/10/01/ipfs-distributed-technology-aids-catalonia-rubber-bullets-fired">referendum in Catalonia</a>, at one point Russia blocked 25% of the Internet because people were saying things the government didn’t like on Telegram.</p>
<p>You should also know that most of the Internet is hosted on Amazon servers. This is another topic, people think that Amazon sells books and toilet brushes, but actually, they sell Internet infrastructure – that’s their core business. And Amazon is an example of a company that doesn’t care about this freedom I mentioned above, they just want to make money. They are not apologetic about it.</p>
<p>So we have been saying for a long time, that the moment you buy into Amazon, the moment you buy into Facebook where it is OK to censor people and trace and track everything, there is no turning back. Governments, companies, and other entities… once they gain such power, they will never give it up.</p>
<p>In opposition to that, people like Juan Benet and people from the hacker community were thinking: OK, so we have torrents, where when you download a file or a film, you also upload it, and there is no official uploader nor downloader and no one can go to a single party and force them to take this file offline… this idea was behind IPFS.</p>
<p>People started to realize: wait, so if we use the same principles we used for torrents, and we use them to make a new kind of Web then censorship will become impossible. And at the same time, you don’t need these big companies anymore.</p>
<p>Imagine that every time someone is viewing the <em>Gangnam Style</em> K-pop video on YouTube it gets downloaded from somewhere on their computer. It has 4,387,208,147 views. Sick amount of data for no reason, it’s the same content transferred again and again.</p>
<p><a href="https://www.youtube.com/watch?v=9bZkp7q19f0" target="_blank">
<img src="https://img.youtube.com/vi/9bZkp7q19f0/default.jpg" alt="Watch the video" width="50%" height="50%" />
</a></p>
<blockquote>
<p>Let’s make some assumptions. The video clocks in at 117 Megabytes. That means (at most) 274,286,340,432 Megabytes, or 274.3 Petabytes of data for the video file alone have been sent since this was published. If we assume a total expense of 1 cent per gigabyte (this would include bandwidth and all of the server costs), $2,742,860 has been spent on distributing this one file so far. <br />
Source: <a href="https://ipfs.io/ipfs/QmNhFJjGcMPqpuYfxL62VVB9528NXqDNMFXiqN5bgFYiZ1/its-time-for-the-permanent-web.html">HTTP is obsolete. It’s time for the distributed, permanent web</a> by kyledrake</p>
</blockquote>
<p>And now we arrived at another advantage of IPFS: If I make a video whose content is not politically correct, for example, for my government and I want to share this with the world, there is no one who could possibly take this down. Also, I don’t need to keep it on my server in my house.</p>
<p><strong>I think we’ve covered some important issues here, each of which would probably lead us to an endless and interesting discussion, but let’s focus on a few points: you started talking about how we are vulnerable online, susceptible to tracking. About censorship, privacy… What else happens when we surf the web?</strong></p>
<p>What’s been happening since 9/11 in the USA and every country aligned with the USA is that certain entities within the governments started to think that it is completely alright to forfeit fundamental human rights in general in the face of adversity, particularly terrorism. Suddenly it was alright to lock up and torture people and consider absolutely everyone a suspect. In the wider spectrum, it means that if you watch something “bad” on the Internet, you are susceptible to blackmail.</p>
<table>
<thead>
<tr>
<th style="text-align: center"><img src="/assets/images/2021-06-03-breaking-the-silent-consent/13105939224_cd1a9956b7_o.jpg" /></th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: center"><em>MEP’s demonstrating support for Edward Snowden, who unveiled the government’s extensive and generally unconstitutional domestic spying programs. <a href="https://www.flickr.com/photos/greensefa/13105939224/in/photostream/">CC BY European Union</a></em></td>
</tr>
</tbody>
</table>
<p>All these companies, like Google, Amazon, Facebook, Apple, etc. are not only obligated to give to any government entity all the information they collected but also to keep their mouth shut about it. Literally, everything gets stored. The stuff you did in the past, what house you want to buy, what car you drive, when you have your period, your consumption patterns … and there is a general acceptance of that mainly because people read too little science fiction. <a href="https://vimeo.com/161183966">Vinay Gupta</a>, a great thinker in many fields, regarding where we are as a technical society, said that the problem is that intellectual leaders of this world, people who studied literature have completely overlooked science fiction.</p>
<p>These leaders, Gupta claims, don’t know how information is propagated on the Internet, they don’t know about some systemic behavior of large centralized systems that’s really very important. A government that spies and knows everything about its own citizens is a different kind of government… We live in a world where AI trained in a specific task has by far exceeded human capacity.</p>
<p><strong>Knowing all that, what are the security goals of IPFS Search, and which have already been achieved?</strong></p>
<p>I think one of the things we are trying to hack is not so much technological, which is also why I have been talking politics this whole interview. I think the goal is to do for search what Wikipedia has done for encyclopedias.</p>
<p>The idea would be to have content discovery outside the platform capitalism domain, which is “We are connecting everybody with everybody, you have to go through me, and every time you go through me, you are paying with your attention, which is the most valuable token.” We want to challenge this model and make the search engine very inexpensive first, by making sure that what we make is very close to what users need and want, and second, by making sure that there is not only one place where all these servers run. We fully expect that as the content we put into our database grows, the log on our database also increases exponentially.</p>
<table>
<thead>
<tr>
<th style="text-align: center"><img src="/assets/images/2021-06-03-breaking-the-silent-consent/4370250237_c69b4265f6_o.png" width="100%" /></th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: center"><em>By: <a href="https://www.flickr.com/photos/opensourceway/4370250237">opensource.com</a></em></td>
</tr>
</tbody>
</table>
<p>But also when people that are putting content online start providing some searches, it becomes a decentralized protocol like Bitcoin, for example. We want to set up an incentive system, possibly backed up by a blockchain, possibly backed by a funky thing called Zero-Knowledge proof, where you can actually make sure that a bunch of people can run a search engine, and they can do it in a way where even if it scales up, even if there are lots of people and some of them don’t play by the rules, you can still get reliable search results. This is our long-term vision.</p>
<p><strong>So again, we have a lot of people involved in the process of development. IPFS Search is based on the idea of a community project.</strong></p>
<p>Yes, we are an open-source project, our model is a bit like Wikipedia, but of course, we currently don’t have an index or a catalogue that people can edit just like that. We would like to have some users’ feedback in our actual search results, but there are some technical problems to solve first, so we prefer to focus on the search for now. As for a community contribution, if you want to change something in the user interface, you want to have a filter or suggest something, you can just propose it via our GitHub repository.</p>
<table>
<thead>
<tr>
<th style="text-align: center"><img src="/assets/images/2021-06-03-breaking-the-silent-consent/Wikipedia_Community_cartoon_-_high_quality.png" /></th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: center"><em>Giulia Forsythe, redrawn by Asiyeh Ghayour <a href="https://commons.wikimedia.org/wiki/File:Wikipedia_Community_cartoon_-_high_quality.png">under CC0 1.0</a></em></td>
</tr>
</tbody>
</table>
<p>We really love it if you want to improve our documentation or contribute ideas, that’s super welcome. But at the same time, we know that at some point we might face various kinds of censorship. And this is why we publicly share the entire index of our search engine. No search engine has ever done that. So we are not open-source only because of our code, but also because of our index. Similar to OpenStreetMap, we have the same license for data. It means that if somebody wants to take us down or censor us, there is nothing in the way of other people to fork us, copy and paste our entire search engine. If they take it and make a better search engine based on ours, the only thing they need to do is to share their improvements and data set. It’s a double-decentralized principle.</p>
<hr />
<blockquote>
<p>📢 Let’s summarize what we have said until now: We are focusing on having a working search engine that other people can copy and paste, and later together with other people we want to make it properly decentralized.</p>
</blockquote>
<hr />
<p><strong>Do you have any particular cooperation on the horizon right now?</strong></p>
<p>Certain niche search engines that target a user who has a bit more knowledge about privacy or is into decentralized Web, such as Brave or DuckDuckGo, are interested in having indexing of decentralized Web.</p>
<p>So, we want to see if we can handle web-scale traffic in order to start such cooperation.</p>
<p><strong>If you were to look back, what happened in the project that is worth mentioning?</strong></p>
<p>Interesting question, if I look back, indeed a lot of things have happened. In the team, our social structures, how we are working together, etc. But what’s visible is the new front-end we launched and a new, better structure, beyond basic usability, that allows us to receive contributions but also work faster, make improvements, be more agile, and closer to users.</p>
<table>
<thead>
<tr>
<th style="text-align: center"><img src="/assets/images/2021-06-03-breaking-the-silent-consent/Untitled 1.png" /></th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: center"><em>On <a href="https://api.ipfs-search.com">https://api.ipfs-search.com</a> you can play freely with our API.</em></td>
</tr>
</tbody>
</table>
<p>One of our goals also, instead of having a normal front end, is to offer our services to the world, so people can integrate our search engine into their own websites. We already have an API, so developers can play with our files, directories, and all data and metadata.</p>
<p>What else has happened is porn. What we noticed quite quickly is that some really weird, but not illegal or frightening, stuff got published on our search engine. That’s become a bit of a problem, because we don’t primarily want to be a porn search engine, haha.</p>
<p><img src="/assets/images/2021-06-03-breaking-the-silent-consent/Untitled 2.png" /></p>
<p>To solve it, we implemented a filter, that you can technically also run in a browser, that is using AI to analyze pictures whether it is porn or not. It doesn’t work perfectly, but we might also get to the point where we improve the model ourselves. But what also came with that is using AI to classify our content, and this means we can use it also to do similar stuff with music, to know the genre and group music together, to navigate between text files, we have. It’s very interesting because AI is coming from a point where it was something abstract, that only Google had the power to use, to publicly available models that you can implement. It all leads us to the internal discussion about applying censorship and becoming evil, but we have found a way: we are only blurring the pictures and giving the users choice to switch the filter off.</p>
<p><strong>What are your and your team’s plans for the future? What do you want to achieve within the next 6 months?</strong></p>
<p>We want to go to The Moon and back, haha.</p>
<p>I think in the next year, because of the way IPFS is increasing its popularity, we are going to start growing exponentially. Or IPFS will fail. This means that someone will do the same thing better, and we will move to theirs, our infrastructure is ready for that.</p>
<table>
<thead>
<tr>
<th style="text-align: center"><img src="/assets/images/2021-06-03-breaking-the-silent-consent/Untitled 3.png" /></th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: center"><em>Over the past year, we’ve more than tripled our index as we’re scaling up to a 50-node cluster.</em></td>
</tr>
</tbody>
</table>
<p>I don’t think that will happen, though, because there are large groups that have been actively supporting IPFS. Also, the NFT ecosystem is running on IPFS, there is a lot going on for them. So, if the amount of available content grows exponentially, it means that we have to grow our infrastructure exponentially. We need to be able to expand and have proper frontend and backend teams to also address more features and check what our users need, and try some solutions.</p>
<p>So far we have been three friends working together, like a hobby that grew out of hand, but now we are looking into starting a company or a foundation. Actually, what we would like to be is something that so far doesn’t exist legally, a social enterprise – an organization that at the same time tries to make money while also guaranteeing certain non-monetary, societal goals. So it will be a personal challenge and also a challenge for us as a team.</p>
<p>It will be a very interesting and tricky year for us.</p>Zuzanna MajerOnline privacy and security are too rarely questioned by ordinary users. Taking them for granted comes from the fact that most people believe they have control over the information they share.NSFW-filter for ipfs-search.com2022-04-18T00:00:00-05:002022-04-18T00:00:00-05:00https://blog.ipfs-search.com/NSFW-f70ee<h2 id="the-problem">The problem</h2>
<p>When we upgraded the frontend for IPFS-search, and while doing so made the graphic content a lot more visible, it became immediately apparent that there was a lot of X-rated material on ipfs, and this made the browsing experience less than pleasant at times. Most search queries turned up at least some imagery of explicit scenes;</p>
<p><sub><em>It happened on a boat last Tuesday.</em></sub></p>
<p><img src="/assets/images/2022-04-18-NSFW- f70ee/Untitled.png" alt="picture" /></p>
<p><sub><em>They are white, and they are in a house. What else do you need to know?</em></sub></p>
<p><img src="/assets/images/2022-04-18-NSFW- f70ee/Untitled 1.png" alt="picture" /></p>
<p><sub><em>Fresh ideas on where to get vegetables and what to do with them.</em></sub></p>
<p><img src="/assets/images/2022-04-18-NSFW- f70ee/Untitled 2.png" alt="picture" /></p>
<p><sub><em>Clearly, the girls on the right are captivated by the scene in the middle.</em></sub></p>
<p><img src="/assets/images/2022-04-18-NSFW- f70ee/Untitled 3.png" alt="picture" /></p>
<h2 id="to-the-rescue-nsfwjs">To the rescue: NSFW.js</h2>
<p>Filtering this out is not a trivial matter. In order to do this properly, you need to classify all content automatically, and for this you need an intelligent system. Fortunately, we found <a href="http://nsfwjs.com">NSFW.js</a>, an open source library that implements an already trained AI model to classify images on nudity and pornographic content and should also work for drawings. The library claims to have 93% accuracy. We made it a priority to integrate this into the search engine.</p>
<p>The AI looks at an image and responds with an estimate classification for five categories: ‘porn’, ‘sexy’, ‘hentai’ (sexually explicit drawings), ‘drawing’ (non-explicit), and ‘neutral’. The estimate comes as a number between 0 and 1, with 1 being absolute certainty that it falls in this category and 0 being absolute certainty that it doesn’t.</p>
<h2 id="architecture">Architecture</h2>
<p>For the architecture, we decided on making a <a href="https://github.com/ipfs-search/nsfw-server">microservice</a> to classify IPFS images. The first idea was to cache the results on IPFS, but after some trial and error it seemed that the benefit did not outweigh the trouble, and we decided to work in stead with a simple server-side cache. While the NSFW.js library is targeted for client-side classification, it was relatively simple to integrate it into a node/express server, with a Nginx reverse proxy with a built-in cache.</p>
<p>The rationale for using a microservice, rather than simply frontend-based, was that this would be able to serve both the search frontend and the search crawlers and/or API; where the crawlers in due time would be able to attach metadata about the classification to the database, the frontend would directly be able to access this information as long as it isn’t (yet) available, and decide on whether/how to display the results from the API.</p>
<h2 id="prototype">Prototype</h2>
<p>For the first iteration, the prototype, we did nothing more than to blur out images in the frontend (using CSS) if they would be classified as “not suitable for work”. A simple toggle-switch, with its setting stored in the browsers’ local storage, would turn the feature on and off. The search frontend would call for each individual image the microservice, and as long as the result was undecided, (either because the request was in-flight or because it returned an error), ‘assume the worst’, i.e., keep the image blurred. We implemented a tooltip message displaying the classification percentages for images in the browser, so we could see what data the assessment was based on.</p>
<p><sub><em>Without blur filter (but pixelated for editorial reasons.)</em></sub></p>
<p><img src="/assets/images/2022-04-18-NSFW- f70ee/Untitled 4.png" alt="picture" /></p>
<p><sub><em>With blur filter enabled.</em></sub></p>
<p><img src="/assets/images/2022-04-18-NSFW- f70ee/Untitled 5.png" alt="picture" /></p>
<p>The reason for doing this on the frontend and not yet in the crawler was to field-test the microservice without committing this information to the database, by being able to see directly which images it blurred and which it didn’t.</p>
<p>The result was already much friendlier search-engine with a lot less obnoxious feel to it. It turned out that the estimation thresholds for the categories ‘sexy’, ‘porn’ and ‘hentai’ need to be very low, around 10-15%, or it started to miss a lot of hits. As would be expected, there were some false positives, and the lower the threshold would be set, the more there are. A few false negatives occur too, but not that many.</p>
<p><sub><em>False positive: Obama eating a strawberry classifies as porn with a certainty of 45%. Maybe it is that look on his face.</em></sub></p>
<p><img src="/assets/images/2022-04-18-NSFW- f70ee/Untitled 6.png" alt="picture" /></p>
<p><sub><em>False positive: This guy, unabashedly exhibiting his banana; 24% certain it is pornography. (It seems that the classifier has a thing for fruit.)</em></sub></p>
<p><img src="/assets/images/2022-04-18-NSFW- f70ee/Untitled 7.png" alt="picture" /></p>
<p><sub><em>False positive: These golden lines classify 45% as porn. No comment. They aren’t even that curvy.</em></sub></p>
<p><img src="/assets/images/2022-04-18-NSFW- f70ee/Untitled 8.png" alt="picture" /></p>
<p><sub><em>False negative: Only 8% certain of pornographic content, which doesn’t meet our (current) threshold. Warning: the image contains product placement.</em></sub></p>
<p><img src="/assets/images/2022-04-18-NSFW- f70ee/Untitled 9.png" alt="picture" /></p>
<p><sub><em>False negative: the classifier is probably thrown off by the letters photoshopped as background layer; it is not a drawing, and it is definitely not neutral.</em></sub></p>
<p><img src="/assets/images/2022-04-18-NSFW- f70ee/Untitled 10.png" alt="picture" /></p>
<p><sub><em>False negative; the internet/IPFS is overflowing with this kind of imaginative artwork. Fortunately, most of it is properly classified by NSFW.js as ‘hentai’, because these cartoons are not for kids.</em></sub></p>
<p><img src="/assets/images/2022-04-18-NSFW- f70ee/Untitled 11.png" alt="picture" /></p>
<p>Altogether, the NSFW-filter prototype worked very well, and because of this, we brought the prototype to production, just so we had a UX we could show with some more confidence to people in general.</p>
<h2 id="backend-integration">Backend integration</h2>
<p>The obvious downsides of having this done solely by the frontend are:</p>
<ul>
<li>you can not add an adult-filter to the search API, and simply not showing the results that surpass the threshold causes weird paging issues (e.g., if all results of a single page have positive NSFW classification, you would see an empty page). The best we could come up with was blurring it, but typically, you don’t want these results at all.</li>
<li>Because IPFS is still pretty slow, the first time classification for new content can take long; after this, the cache takes care of it.</li>
</ul>
<p>So, the second phase was to make the code a bit more mature, and incorporate a connection with the microservice into the backend, the crawler. We did this by adding the classifications of files to the metadata of the search engine database. Then the API could filter on it by request.</p>
<p>To do this, we needed to add one more feature: information about which exact AI model had been used for a specific classification. NSFW.js has several models directly available, and it can not be ruled out that other, better ones will become available in the future, or even that we would be training our own datasets e.g. using user feedback. <br />
So, stored data should have a reference to which model was used to generate it, in the way that some next generation API can make informed decisions about, for example, whether to access the microservice for newer data or not. We solved this by calculating the IPFS-CID of the model files (using <a href="https://github.com/ipfs/js-ipfs/tree/master/docs">js-ipfs</a>) and adding this to the classification-microservices’ output.</p>
<p>Finally, we integrated the microservice API into the crawler and added a nsfw filter on the frontend for the search query. It was notable that images that had been indexed before the nsfw-microservice had been connected to the crawler were omitted from filtered results, as could be expected.</p>
<h2 id="considerations-and-debate">Considerations and debate</h2>
<p>It is currently unknown to us how the 93% accuracy has been calculated, but with any AI based classification, you will always get a number of false positives and negatives. We considered using user feedback for improving the model, but quickly abandoned this because of all the complications this would bring. There are GDPR regulations, storage of feedback data, fighting trolls and bots and trollbots, design of UX for feedback, security, QA, and so forth. But most of all was there the already tough issue of keeping websearch neutral, unbiased and completely private while at the same time having to curate users’ opinions about sensitive, highly debatable matters.</p>
<p>Because the debate does not end with filtering nudity, it merely starts there. What about targeted violence, fake-news, controversial symbols or politics, discrimination, etc. etc.? What about written documents or audiorecordings, shouldn’t these be filtered? With the resources we have now, this is too much to be dealing with, and it may not be urgent, yet. However, with an increasing user base and search-index covering more and more materials, these questions are likely to come up down the line. A good set of solutions solution to deal with this will obviously be much more complex than implementing an open source library into the system.</p>
<h2 id="bonus-bonus">Bonus bonus!</h2>
<p>As the nsfw filter classifies for drawings too, we can use it to create a query parameter filter for that too, without much effort.</p>
<h2 id="conclusion">Conclusion</h2>
<p>We were successful to deal with the issue of content that is ‘not suitable for work’ in a straightforward way without the need for too many resources, thanks to the plugin <a href="https://nsfwjs.com/">NSFW.js</a>. The user experience of <a href="http://IPFS-search.com">IPFS-search.com</a> has increased a lot as a consequence.</p>
<h2 id="references">References</h2>
<ol>
<li>Microservice repo - <a href="https://github.com/ipfs-search/nsfw-server">https://github.com/ipfs-search/nsfw-server</a></li>
<li>NSFW.js - <a href="https://nsfwjs.com/">https://nsfwjs.com/</a></li>
<li><a href="https://ipfs-search.com/">ipfs-search.com</a></li>
</ol>Frido EmansTo make searching more pleasant on IPFS-search.com, we implemented a filter for X-rated content.Making ipfs-search distributed2021-09-24T00:00:00-05:002021-09-24T00:00:00-05:00https://blog.ipfs-search.com/making-ipfs-search-distributed<h1 id="a-sketch-for-how-distributed-search-could-be-realized-for-the-ipfs-and-the-distributed-web">A sketch for how distributed search could be realized for the IPFS and the distributed web</h1>
<p><strong><em>Special thanks to Nina for creating this sketch</em></strong></p>
<h3 id="how-we-could-realize-distributed-search">How we could realize distributed search:</h3>
<ul>
<li>Provider nodes that wish to participate, parse and index only the files they have added to a dweb (DHT hashes) and that have world file permissions.</li>
<li>This local index is put on an (IPFS) cluster.</li>
<li>A query can use the distributed index.</li>
<li>Initial search functionality is a basic boolean search.</li>
<li>Settings functionality anticipates tuning.</li>
<li>In the future, one can add to the search engine functionality with extensions.</li>
</ul>
<p>Control for users.</p>
<h2 id="sketch">Sketch</h2>
<p>The ballon d’essai can consist of:</p>
<ul>
<li>A <a href="https://cluster.ipfs.io/">distributed index using an IPFS cluster</a></li>
<li>An indexer package with which content providers can index what they provide and add such an index to the distributed index, starting with indexing documents</li>
<li>A thin, separate client with which people can query the distributed index and receive results ranked relevant to the query.</li>
</ul>
<h3 id="overlay-networks">Overlay networks</h3>
<p>An IPFS node can be fingerprinted through the content it stores. An overlay network needs to offer an “anonymous” mode that only enables features known to not leak information.</p>
<ul>
<li>No local discovery.</li>
<li>No transports other than, for example, via Tor (an overlay network consisting of more than seven thousand relays to conceal a user’s location and usage from anyone conducting network surveillance or traffic analysis).</li>
<li>Private routing to make the network non-enumerable.</li>
</ul>
<h3 id="parsing">Parsing</h3>
<p>We could code different parsers for each type of file but that is not our main focus at the moment, and because a Python port of the Apache Tika library exists that according to the documentation supports text extraction from over 1500 file formats, we go with that, at least for now. But it is slow, and in the future we may reconsider.</p>
<p>This parser is pointed to the root of a site or a collection, parses its content (thereby creating a corpus) and adds the objects to ipfs, rather than fetching the ipfs hashtable and taking it from there. Again, we wish to focus on the indexing and clustering of a distributed index, not on finding out how to use the ipfs hashtable (for now).</p>
<p><em>Code on this page is a first shot and should be read as pseudocode snippets.</em></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>import os, os.path
import ipfsApi
from tika import parser
from multiprocessing import Pool
def tika_parser(file_path):
# Extract text from document
content = parser.from_file(file_path)
if 'content' in content:
text = content['content']
else:
return
# Convert to string
text = str(text)
# Normalisation to utf-8 format
safe_text = text.encode('utf-8', errors='ignore')
# Escape any \ issues
safe_text = str(safe_text).replace('\\', '\\\\').replace('"', '\\"')
# Add hash (as filename) and content of file to corpus dataframe
...
def walkthrough ()
corpus_root = os.getxxx (path_to_root)
walk through the directory structure to fetch each file_path and
add each encountered object to ipfs (if duplicate, will not be pinned)
add hash and file_path to paths
return paths
pool = Pool()
pool.map(tika_parser, paths)
return corpus
</code></pre></div></div>
<h3 id="resources">Resources</h3>
<ul>
<li><a href="https://tika.apache.org/1.4/formats.html" title="https://tika.apache.org/1.4/formats.html">Tika Supported Document Formats</a></li>
<li><a href="https://pypi.org/project/ipfs-api/" title="https://pypi.org/project/ipfs-api/">IPFS API Bindings for Python</a></li>
</ul>
<h2 id="distributing-the-index-on-an-ipfs-cluster">Distributing the index on an IPFS Cluster</h2>
<ul>
<li>IPFS does not guarantee redundancy. We can use IPFS clustering.</li>
<li>Only popular indexes will be able to get a decent speed.
<ul>
<li>We can run a few web agent type ipfs nodes in a cluster that pin all the indexes. Give these enough bandwidth and we have some basis nodes that can act as mirrors and can also be served via HTTPS (the internet-facing demo version).</li>
<li>IPFS can replace mirror indexes with IPNS addresses. We will still need reliable hosting for these initial seeders.</li>
</ul>
</li>
</ul>
<h2 id="risks">Risks</h2>
<p>IPFS is still in alpha development. That means there are a lot of (undiscovered) bugs and vulnerabilities and the code is not stable. This could create (security) problems.</p>
<h3 id="resources-1">Resources</h3>
<ul>
<li><a href="https://github.com/ipfs/ipfs-cluster/" title="https://github.com/ipfs/ipfs-cluster/">IPFS Cluster Github</a></li>
<li><a href="https://cluster.ipfs.io/documentation/" title="https://cluster.ipfs.io/documentation/">IPFS Cluster Documentation</a></li>
<li><a href="https://cluster.ipfs.io/documentation/deployment/architecture/" title="https://cluster.ipfs.io/documentation/deployment/architecture/">IPFS Cluster Architecture overview</a></li>
</ul>
<h2 id="querying-the-index">Querying the index</h2>
<p>Our intention is to support boolean queries and phrase queries.</p>
<ul>
<li>Sanitize the query (stemming all the words, making all letters lowercase, removing punctuation)</li>
<li>Tokenise the query (split into words)</li>
<li>Get term lists from the distributed index, which documents they appear in, and union the lists</li>
</ul>
<h3 id="boolean-query">Boolean query</h3>
<p>For each inverted index from self and received from neighbours:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>def one_word_query(word, invertedIndex):
pattern = re.compile('[\W_]+')
word = pattern.sub(' ',word)
if word in invertedIndex.keys():
return [filename for filename in invertedIndex[word].keys()]
else:
return []
</code></pre></div></div>
<p><strong>OR</strong></p>
<p><strong>Aggregate lists and union</strong></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>def free_text_query(string):
pattern = re.compile('[\W_]+')
string = pattern.sub(' ',string)
result = []
for word in string.split():
result += one_word_query(word)
return list(set(result))
</code></pre></div></div>
<p><strong>AND</strong></p>
<p>For an AND use an intersection instead of a union to aggregate the results of the single word queries.</p>
<h3 id="phrase-query">Phrase query</h3>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>def phrase_query(string, invertedIndex):
pattern = re.compile('[\W_]+')
string = pattern.sub(' ',string)
listOfLists, result = [],[]
for word in string.split():
listOfLists.append(one_word_query(word))
setted = set(listOfLists[0]).intersection(*listOfLists)
for filename in setted:
temp = []
for word in string.split():
temp.append(invertedIndex[word][filename][:])
for i in range(len(temp)):
for ind in range(len(temp[i])):
temp[i][ind] -= i
if set(temp[0]).intersection(*temp):
result.append(filename)
return rankResults(result, string)
</code></pre></div></div>
<h2 id="yggdrasil">Yggdrasil</h2>
<p>Yggdrasil is an early-stage implementation of a fully end-to-end encrypted IPv6 network. It is lightweight, self-arranging, supported on multiple platforms and allows pretty much any IPv6-capable application to communicate securely with other Yggdrasil nodes. Yggdrasil does not require IPv6 Internet connectivity - it also works over IPv4.</p>
<p>Looking at it for its clustering and bootstrapping implementation.</p>
<h3 id="resources-2">Resources</h3>
<ul>
<li><a href="https://yggdrasil-network.github.io/">Yggdrasil</a></li>
<li><a href="https://github.com/yggdrasil-network/yggdrasil-go">Yggdrasil Github</a></li>
</ul>
<h2 id="testing">Testing</h2>
<ul>
<li>Scalability indicators
<ul>
<li>Number of hashes crawled per second per-peer versus the number of peers</li>
<li>Number of downloaded bytes per second versus the number of peers</li>
</ul>
</li>
<li>Performance indicators
<ul>
<li>Number of hashes crawled per second versus different CPU loads/platforms</li>
<li>Throughput of a peer versus the number of crawled job queues (to determine the optimal number of crawl job queues) per platform (differentiate using agent attributes).</li>
</ul>
</li>
<li>Node failure
<ul>
<li>If automated, this may require adding data entry points in the API that are only used for testing.</li>
<li>Add test data, check that it has been added and has propagated throughout the neighbourhood.</li>
<li>Take an agent offline (check that it has gone down and is inaccessible) and verify that all the data appears to be working.</li>
<li>Pull data manually from each data store (check there are no errors as a result) on the agent, and verify that the data is still retrievable from the system.</li>
<li>Bring the downed node back online. The data that belongs on this node begins to flow back into the node.</li>
<li>After a while, pull the data from the agent to check that data that was sent to its neighbours when it was down is stored correctly.</li>
</ul>
</li>
<li>Predictive analysis
<ul>
<li>Test for false negatives and false positives of the various classifiers with unlabelled traffic data[^testing]</li>
</ul>
</li>
</ul>
<h3 id="resources-3">Resources</h3>
<ul>
<li><a href="https://www.fed4fire.eu/testbeds/grid5000/" title="https://www.fed4fire.eu/testbeds/grid5000/">Grid’5000</a></li>
</ul>ipfs-search.comA sketch of how it could be done