Buckets:

download
raw
11.5 kB
import"../chunks/DsnmJJEf.js";import{i as V,h as J,C as Y,H as _,a as w,D as s,E as M,s as X}from"../chunks/Y07aUP_m.js";import{p as Z,o as j,s as e,f as Q,a as F,b as z,d as a,c as I,n as t,r as o}from"../chunks/DEl054mf.js";const K='{"title":"Utilities","local":"utilities","sections":[{"title":"Configure logging","local":"datasets.utils.logging.get_verbosity","sections":[],"depth":2},{"title":"Configure progress bars","local":"datasets.enable_progress_bars","sections":[],"depth":2}],"depth":1}';var $=I('<meta name="hf:doc:metadata"/>'),ee=I(`<p></p> <!> <!> <!> <p>🤗 Datasets strives to be transparent and explicit about how it works, but this can be quite verbose at times. We have included a series of logging methods which allow you to easily adjust the level of verbosity of the entire library. Currently the default verbosity of the library is set to <code>WARNING</code>.</p> <p>To change the level of verbosity, use one of the direct setters. For instance, here is how to change the verbosity to the <code>INFO</code> level:</p> <!> <p>You can also use the environment variable <code>DATASETS_VERBOSITY</code> to override the default verbosity, and set it to one of the following: <code>debug</code>, <code>info</code>, <code>warning</code>, <code>error</code>, <code>critical</code>:</p> <!> <p>All the methods of this logging module are documented below. The main ones are:</p> <ul><li><a href="/docs/datasets/pr_8864/en/package_reference/utilities#datasets.utils.logging.get_verbosity">logging.get_verbosity()</a> to get the current level of verbosity in the logger</li> <li><a href="/docs/datasets/pr_8864/en/package_reference/utilities#datasets.utils.logging.set_verbosity">logging.set_verbosity()</a> to set the verbosity to the level of your choice</li></ul> <p>In order from the least to the most verbose (with their corresponding <code>int</code> values):</p> <ol><li><code>logging.CRITICAL</code> or <code>logging.FATAL</code> (int value, 50): only report the most critical errors.</li> <li><code>logging.ERROR</code> (int value, 40): only report errors.</li> <li><code>logging.WARNING</code> or <code>logging.WARN</code> (int value, 30): only reports error and warnings. This the default level used by the library.</li> <li><code>logging.INFO</code> (int value, 20): reports error, warnings and basic information.</li> <li><code>logging.DEBUG</code> (int value, 10): report all information.</li></ol> <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"><!> <p>Return the current level for the HuggingFace datasets library’s root logger.</p> <blockquote class="tip"><p>HuggingFace datasets library has following logging levels:</p> <ul><li><code>datasets.logging.CRITICAL</code>, <code>datasets.logging.FATAL</code></li> <li><code>datasets.logging.ERROR</code></li> <li><code>datasets.logging.WARNING</code>, <code>datasets.logging.WARN</code></li> <li><code>datasets.logging.INFO</code></li> <li><code>datasets.logging.DEBUG</code></li></ul></blockquote></div> <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"><!> <p>Set the level for the Hugging Face Datasets library’s root logger.</p></div> <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"><!> <p>Set the level for the Hugging Face datasets library’s root logger to <code>INFO</code>.</p> <p>This will display most of the logging information and tqdm bars.</p> <p>Shortcut to <code>datasets.logging.set_verbosity(datasets.logging.INFO)</code>.</p></div> <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"><!> <p>Set the level for the Hugging Face datasets library’s root logger to <code>WARNING</code>.</p> <p>This will display only the warning and errors logging information and tqdm bars.</p> <p>Shortcut to <code>datasets.logging.set_verbosity(datasets.logging.WARNING)</code>.</p></div> <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"><!> <p>Set the level for the Hugging Face datasets library’s root logger to <code>DEBUG</code>.</p> <p>This will display all the logging information and tqdm bars.</p> <p>Shortcut to <code>datasets.logging.set_verbosity(datasets.logging.DEBUG)</code>.</p></div> <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"><!> <p>Set the level for the Hugging Face datasets library’s root logger to <code>ERROR</code>.</p> <p>This will display only the errors logging information and tqdm bars.</p> <p>Shortcut to <code>datasets.logging.set_verbosity(datasets.logging.ERROR)</code>.</p></div> <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"><!> <p>Disable propagation of the library log outputs.
Note that log propagation is disabled by default.</p></div> <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"><!> <p>Enable propagation of the library log outputs.
Please disable the Hugging Face datasets library’s default handler to prevent double logging if the root logger has
been configured.</p></div> <!> <p>By default, <code>tqdm</code> progress bars will be displayed during dataset download and preprocessing. You can disable them globally by setting <code>HF_DATASETS_DISABLE_PROGRESS_BARS</code> environment variable. You can also enable/disable them using <a href="/docs/datasets/pr_8864/en/package_reference/utilities#datasets.enable_progress_bars">enable_progress_bars()</a> and <a href="/docs/datasets/pr_8864/en/package_reference/utilities#datasets.disable_progress_bars">disable_progress_bars()</a>. If set, the environment variable has priority on the helpers.</p> <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"><!> <p>Enable globally progress bars used in <code>datasets</code> except if <code>HF_DATASETS_DISABLE_PROGRESS_BARS</code> environment
variable has been set.</p> <p>Use <a href="/docs/datasets/pr_8864/en/package_reference/utilities#datasets.disable_progress_bars">disable_progress_bars()</a> to disable them.</p></div> <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"><!> <p>Disable globally progress bars used in <code>datasets</code> except if <code>HF_DATASETS_DISABLE_PROGRESS_BARS</code> environment
variable has been set.</p> <p>Use <a href="/docs/datasets/pr_8864/en/package_reference/utilities#datasets.enable_progress_bars">enable_progress_bars()</a> to re-enable them.</p></div> <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"><!> <p>Return whether progress bars are globally disabled or not.</p> <p>Progress bars used in <code>datasets</code> can be enable or disabled globally using <a href="/docs/datasets/pr_8864/en/package_reference/utilities#datasets.enable_progress_bars">enable_progress_bars()</a> and <a href="/docs/datasets/pr_8864/en/package_reference/utilities#datasets.disable_progress_bars">disable_progress_bars()</a> or by setting <code>HF_DATASETS_DISABLE_PROGRESS_BARS</code> as environment variable.</p></div> <!> <p></p>`,1);function oe(L,N){Z(N,!1),j(()=>{new URLSearchParams(window.location.search).get("fw")}),V();var v=ee();J("m2ngq6",E=>{var T=$();X(T,"content",K),F(E,T)});var m=e(Q(v),2);Y(m,{containerStyle:"float: right; margin-left: 10px; display: inline-flex; position: relative; z-index: 10;"});var y=e(m,2);_(y,{title:"Utilities",local:"utilities",headingTag:"h1"});var f=e(y,2);_(f,{title:"Configure logging",local:"datasets.utils.logging.get_verbosity",headingTag:"h2"});var S=e(f,6);w(S,{code:"aW1wb3J0JTIwZGF0YXNldHMlMEFkYXRhc2V0cy5sb2dnaW5nLnNldF92ZXJib3NpdHlfaW5mbygp",highlighted:`<span class="hljs-keyword">import</span> datasets
datasets.logging.set_verbosity_info()`,lang:"py",wrap:!1});var R=e(S,4);w(R,{code:"REFUQVNFVFNfVkVSQk9TSVRZJTNEZXJyb3IlMjAuJTJGbXlwcm9ncmFtLnB5",highlighted:"DATASETS_VERBOSITY=error ./myprogram.py",lang:"bash",wrap:!1});var r=e(R,10),x=a(r);s(x,{name:"datasets.utils.logging.get_verbosity",anchor:"datasets.utils.logging.get_verbosity",source:"https://github.com/huggingface/datasets/blob/r_8864/src/datasets/utils/logging.py#L94",parameters:[],returnDescription:`<script context="module">export const metadata = 'undefined';<\/script>
<p>Logging level, e.g., <code>datasets.logging.DEBUG</code> and <code>datasets.logging.INFO</code>.</p>
`}),t(4),o(r);var i=e(r,2),D=a(i);s(D,{name:"datasets.utils.logging.set_verbosity",anchor:"datasets.utils.logging.set_verbosity",source:"https://github.com/huggingface/datasets/blob/r_8864/src/datasets/utils/logging.py#L110",parameters:[{name:"verbosity",val:": int"}],parametersDescription:[{anchor:"datasets.utils.logging.set_verbosity.verbosity",description:`<strong>verbosity</strong> &#x2014;
Logging level, e.g., <code>datasets.logging.DEBUG</code> and <code>datasets.logging.INFO</code>.`,name:"verbosity"}]}),t(2),o(i);var l=e(i,2),B=a(l);s(B,{name:"datasets.utils.logging.set_verbosity_info",anchor:"datasets.utils.logging.set_verbosity_info",source:"https://github.com/huggingface/datasets/blob/r_8864/src/datasets/utils/logging.py#L119",parameters:[]}),t(6),o(l);var g=e(l,2),G=a(g);s(G,{name:"datasets.utils.logging.set_verbosity_warning",anchor:"datasets.utils.logging.set_verbosity_warning",source:"https://github.com/huggingface/datasets/blob/r_8864/src/datasets/utils/logging.py#L129",parameters:[]}),t(6),o(g);var d=e(g,2),O=a(d);s(O,{name:"datasets.utils.logging.set_verbosity_debug",anchor:"datasets.utils.logging.set_verbosity_debug",source:"https://github.com/huggingface/datasets/blob/r_8864/src/datasets/utils/logging.py#L139",parameters:[]}),t(6),o(d);var n=e(d,2),k=a(n);s(k,{name:"datasets.utils.logging.set_verbosity_error",anchor:"datasets.utils.logging.set_verbosity_error",source:"https://github.com/huggingface/datasets/blob/r_8864/src/datasets/utils/logging.py#L149",parameters:[]}),t(6),o(n);var c=e(n,2),H=a(c);s(H,{name:"datasets.utils.logging.disable_propagation",anchor:"datasets.utils.logging.disable_propagation",source:"https://github.com/huggingface/datasets/blob/r_8864/src/datasets/utils/logging.py#L159",parameters:[]}),t(2),o(c);var b=e(c,2),q=a(b);s(q,{name:"datasets.utils.logging.enable_propagation",anchor:"datasets.utils.logging.enable_propagation",source:"https://github.com/huggingface/datasets/blob/r_8864/src/datasets/utils/logging.py#L166",parameters:[]}),t(2),o(b);var A=e(b,2);_(A,{title:"Configure progress bars",local:"datasets.enable_progress_bars",headingTag:"h2"});var p=e(A,4),C=a(p);s(C,{name:"datasets.enable_progress_bars",anchor:"datasets.enable_progress_bars",source:"https://github.com/huggingface/datasets/blob/r_8864/src/datasets/utils/tqdm.py#L78",parameters:[]}),t(4),o(p);var u=e(p,2),U=a(u);s(U,{name:"datasets.disable_progress_bars",anchor:"datasets.disable_progress_bars",source:"https://github.com/huggingface/datasets/blob/r_8864/src/datasets/utils/tqdm.py#L61",parameters:[]}),t(4),o(u);var h=e(u,2),W=a(h);s(W,{name:"datasets.are_progress_bars_disabled",anchor:"datasets.are_progress_bars_disabled",source:"https://github.com/huggingface/datasets/blob/r_8864/src/datasets/utils/tqdm.py#L95",parameters:[]}),t(4),o(h);var P=e(h,2);M(P,{source:"https://github.com/huggingface/datasets/blob/main/docs/source/package_reference/utilities.mdx"}),t(2),F(L,v),z()}export{oe as component};

Xet Storage Details

Size:
11.5 kB
·
Xet hash:
14e646f12004a4240ec1d4cebc5baebe70d60bc65f5deb6b6cd0a120290fbc5e

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.