Commit History

  1. :zap: [Enhance] Add ignore classes for wikipedia.org

    dd996c1
  2. :zap: [Enhance] Loop multiple conditions for extracting abstract

    a315628
  3. :boom: [Fix] PermissionError: [Errno 13] Permission denied: /app/files

    ea49401
  4. :gem: [Feature] Sync to Hugginface space, and run in docker

    d0a4f07
  5. :boom: [Fix] SearchAPIApp: incorrect order of extracted contents to urls

    eb0ce75
  6. :pencil: [Config] Generate requirements.txt

    d3368e0
  7. :recycle: [Refactor] Replace output_path with html_path to avoid confuse

    8c0b736
  8. :gem: [Feature] SearchAPIApp: Concurrent fetch urls and extract contents

    f234ce3
  9. :boom: [Fix] WebpageFetcher: raise timeout when request.get hangs

    bce51d4
  10. :boom: [Fix] WebpageContentExtractor: UnicodeDecodeError

    cff1afc
  11. :recycle: [Refactor] QueryResultsExtractor: prettify logging

    0acc824
  12. :zap: [Enhance] BatchWebpageFetcher: return url_and_output_path_list

    4591d96
  13. :gem: [Feature] New BatchWebpageContentExtractor: Extract webpage content from multiple html_paths concurrently

    1db460d
  14. :gem: [Feature] New BatchWebpageFetcher: Fetch multiple urls concurrently

    e92817a
  15. :boom: [Fix] Duplicated query_results in response JSON when passing multiple queries

    876e441

  1. :zap: [Enhance] WebpageContentExtractor: Escape dash, and ignore⁠

    c7c538d
  2. :zap: [Enhance] ignore classes pattern, especially for 163.com

    3dda344
  3. :zap: [Enhance] Rename HTMLFetcher to WebpageFetcher, and add output_parent param

    62ee9e4
  4. :zap: [Enhance] SearchAPIApp: overwrite param for query and webpage HTML

    9fb4731
  5. :recycle: [Refactor] WebpageContentExtractor: Separate html and markdown processing

    a636bcb
  6. :recycle: [Refactor] Move hardcoded consts to network_configs

    af2c647
  7. :zap: [Enhance] HTMLFetcher and GoogleSearcher: support cache with overwrite, and ignore host

    cf4c3f8
  8. :gem: [Feature] SearchAPIApp: New extract_content param

    4d3e890
  9. :gem: [Feature] New WebpageContentExtractor: extract webpage content as clean markdown

    e773696
  10. :recycle: [Refactor] HTMLFetcher: replace save_path with output_path

    7d44e75
  11. :gem: [Feature] Enable SearchAPIApp: /queries_to_search_results

    138c09e
  12. :zap: [Enhance] GoogleSearcher: Add params of result_sum and safe

    8bf48d8
  13. :recycle: [Refactor] Rename SearchResultsExtractor to QueryResultsExtractor, and store results

    0f6452f
  14. :zap: [Enhance] FilepathConverter: New parent param when init

    f9c42cf

  1. :gem: [Feature] New HTMLFetcher: download url to local html file

    b259fec
  2. :gem: [Feature] New FilepathConverter: convert urls and queries to valid file path

    64a0dbf

  1. :recycle: [Refactor] Move header constructor, and prettier logging

    e448a74
  2. :gem: [Feature] SearchResultsExtractor: related questions

    f150f6b
  3. :gem: [Feature] New SearchResultsExtractor: title, site, link, abstract

    ef3de03
  4. :pencil: [Doc] Readme and git ignore

    d6015f4
  5. :gem: [Feature] New Enver and Logger

    f2ec1a1
  6. :gem: [Feature] New GoogleSearcher: Enable google search with query

    6cf0820