fix(app): collection fetches addresses with Korean paths and words

The request line is ASCII: other characters and spaces in the path and query go as UTF-8 %HH,
while existing %HH and separators stay.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
javis-bot
2026-11-13 21:49:01 +09:00
parent d0cb540fa0
commit 69f1bf4fa9
3 changed files with 16 additions and 2 deletions

View File

@@ -135,6 +135,10 @@ def _connect(ips: list, port: int, deadline: float) -> socket.socket:
raise
def _ascii(part: str) -> str:
return urllib.parse.quote(part, safe="/%:@!$&'()*+,;=?[]~")
def _get(url: str, deadline: float, dog: _Watchdog):
"""One hop: ("redirect", next url) or ("body", bytes)."""
import http.client
@@ -161,7 +165,9 @@ def _get(url: str, deadline: float, dog: _Watchdog):
else:
conn = http.client.HTTPConnection(p.hostname, port)
conn.sock = sock
path = (p.path or "/") + (f"?{p.query}" if p.query else "")
# The request line is ASCII: other characters (and spaces) go as UTF-8 %HH; existing %HH
# and the reserved separators stay as they are.
path = _ascii(p.path or "/") + (f"?{_ascii(p.query)}" if p.query else "")
conn.request("GET", path, headers={"User-Agent": USER_AGENT, "Accept-Encoding": "identity"})
r = conn.getresponse()
if r.status in (301, 302, 303, 307, 308):

View File

@@ -338,7 +338,8 @@ still deleted (below).
local network. Every redirect is checked the same way. At most 1 MB and 15 s in all: the name lookup,
connecting, TLS, the headers and the body of every redirect count, and a server that sends
slowly, even byte by byte within the per-read limit, is cut off at 15 s. A response cut short is a
failure. The check logs nothing about the address. The collected text is web
failure. Characters outside ASCII (and spaces) in the path and words go as UTF-8 `%HH`; existing
`%HH` and separators stay. The check logs nothing about the address. The collected text is web
content: it goes only to `<dir>/<name>/<YYYY-MM-DD>.jsonl` (one JSON object per line: `at`, then
`title`, `url`, `text` as the kind has them), never to the brain or a log.
- Day files older than `keepDays` are deleted in every folder of `dir` at each check, also of