@RL_Dane This wasn't done to extract every bit out of joy in life, but in response to a flood of scraping that caused three outages, taking down the site for everyone. Hopefully it's temporary.

https://discuss.python.org/t/fastly-interfering-with-pypi-search/73597/6

The JS issue has been reported here:
https://github.com/pypi/warehouse/issues/17285
(I see that GitHub search also doesn't work without JS.)

The XMLRPC search feature was also unfortunately removed after years of abuse:

https://github.com/pypi/warehouse/issues/9136#issuecomment-790598298

#Python #PyPI

@hugovk

Ah, that's awful :(

Even if they just provided something like a downloadable daily export of the list, that would be enough. I just hate to have to use a web browser for everything.

@miketheman @hugovk

Man, I can't imagine snarfing an 29 MiB html file being easier on the server than a simple query, but this will do very nicely for my purposes, thank you. I wish it had a brief description as well, but it's totally better than nothing.

I'll definitely cache it for a week or so in my script so I'm not grabbing it each time I do a search.

Just in case anyone's curious, the script I use is here: https://codeberg.org/rldane/scripts/src/branch/main/searchall
It's the "-p" option.

@RL_Dane @hugovk The index is heavily cached at the edge, so there's no impact on server side. Since search is effectively an unbounded context, the backend gets called every time.
Glad to see you were able to make it work!

@miketheman @hugovk

Thanks so much!

If you find out about a variant that has short descriptions like:

`pelican - Static site generator supporting Markdown and reStructuredText`

Please let me know! Thanks!! <3