simplyc6 commited on
Commit
60beac4
·
verified ·
1 Parent(s): 56b2b6e

Create a web-based audio visualizer that generates immersive, procedurally created visuals based on uploaded audio files. The system should leverage medically backed data on synesthesia, auditory perception, and cross-modal sensory integration to simulate how people with synesthesia may “see” music.

Browse files

Core Requirements:

Audio Input & Processing

Users can upload .wav or .mp3 files.

The engine extracts frequency bands, amplitude, tempo, and timbral qualities.

Use FFT or similar techniques to generate real-time analysis.

Medical/Scientific Backing

Base color, motion, and particle mapping rules on published neurological and psychological studies of synesthesia (e.g., sound-color, pitch-shape, rhythm-motion correlations).

Allow configurable “synesthesia modes” such as Chromesthesia (sound → color), Auditory-Tactile (sound → vibration patterns), etc.

Visualization Engine

Render visuals in WebGL (Three.js or similar) for smooth, GPU-accelerated performance.

Procedurally generate glowing particle fields, fluid motions, and geometric transformations tied to music intensity and frequency.

Ensure visuals evolve dynamically over time, not just repeating loops.

UI/UX

Simple upload interface for audio files.

Play/Pause controls with timeline scrubbing.

Option to toggle different visualization “modes” (scientifically inspired presets).

Responsive design so it works on desktop and mobile browsers.

Extendability

Architecture should allow adding new medically informed mappings later (e.g., linking timbre → texture, harmony → spatial depth).

Export visuals as high-resolution video or shareable web links.

Deliverables:

A web application prototype (React + Three.js or similar) hosted in the browser.

A modular procedural visualization engine that maps sound features to medically grounded sensory equivalents.

Clean, well-documented code so mappings can be extended with new research data. - Initial Deployment

Files changed (3) hide show
  1. README.md +7 -5
  2. index.html +358 -18
  3. prompts.txt +49 -0
README.md CHANGED
@@ -1,10 +1,12 @@
1
  ---
2
- title: Synesthesia
3
- emoji: 🌖
4
- colorFrom: yellow
5
- colorTo: pink
6
  sdk: static
7
  pinned: false
 
 
8
  ---
9
 
10
- Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
 
1
  ---
2
+ title: synesthesia
3
+ emoji: 🐳
4
+ colorFrom: gray
5
+ colorTo: green
6
  sdk: static
7
  pinned: false
8
+ tags:
9
+ - deepsite
10
  ---
11
 
12
+ Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
index.html CHANGED
@@ -1,19 +1,359 @@
1
- <!doctype html>
2
- <html>
3
- <head>
4
- <meta charset="utf-8" />
5
- <meta name="viewport" content="width=device-width" />
6
- <title>My static Space</title>
7
- <link rel="stylesheet" href="style.css" />
8
- </head>
9
- <body>
10
- <div class="card">
11
- <h1>Welcome to your static Space!</h1>
12
- <p>You can modify this app directly by editing <i>index.html</i> in the Files and versions tab.</p>
13
- <p>
14
- Also don't forget to check the
15
- <a href="https://huggingface.co/docs/hub/spaces" target="_blank">Spaces documentation</a>.
16
- </p>
17
- </div>
18
- </body>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
19
  </html>
 
1
+ <!DOCTYPE html>
2
+ <html lang="en">
3
+ <head>
4
+ <meta charset="UTF-8">
5
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
6
+ <title>Synesthetic Audio Visualizer</title>
7
+ <script src="https://cdn.tailwindcss.com"></script>
8
+ <script src="https://cdnjs.cloudflare.com/ajax/libs/three.js/r128/three.min.js"></script>
9
+ <script src="https://cdn.jsdelivr.net/npm/vanta@latest/dist/vanta.globe.min.js"></script>
10
+ <script src="https://unpkg.com/feather-icons"></script>
11
+ <script src="https://cdn.jsdelivr.net/npm/p5@1.4.0/lib/p5.js"></script>
12
+ <style>
13
+ #audioCanvas {
14
+ position: absolute;
15
+ top: 0;
16
+ left: 0;
17
+ width: 100%;
18
+ height: 100%;
19
+ z-index: -1;
20
+ }
21
+ .file-upload:hover {
22
+ transform: translateY(-2px);
23
+ box-shadow: 0 10px 20px rgba(0, 0, 0, 0.2);
24
+ }
25
+ .synesthesia-mode:hover {
26
+ transform: scale(1.05);
27
+ transition: all 0.3s ease;
28
+ }
29
+ </style>
30
+ </head>
31
+ <body class="bg-gray-900 text-white min-h-screen flex flex-col">
32
+ <div id="vanta-bg" class="fixed inset-0 opacity-20"></div>
33
+
34
+ <header class="py-6 px-4 sm:px-6 lg:px-8 text-center">
35
+ <h1 class="text-4xl md:text-5xl font-bold mb-2 bg-clip-text text-transparent bg-gradient-to-r from-purple-400 via-pink-500 to-red-500">
36
+ Synesthetic Audio Visualizer
37
+ </h1>
38
+ <p class="text-lg text-gray-300 max-w-2xl mx-auto">
39
+ Experience music through the lens of synesthesia. Scientifically grounded audio-visual mappings.
40
+ </p>
41
+ </header>
42
+
43
+ <main class="flex-grow container mx-auto px-4 py-8 flex flex-col items-center">
44
+ <div class="w-full max-w-2xl bg-gray-800 bg-opacity-70 backdrop-blur-lg rounded-xl shadow-2xl p-6 mb-8">
45
+ <div class="flex flex-col sm:flex-row gap-4 mb-6">
46
+ <label class="file-upload flex-1 cursor-pointer">
47
+ <input type="file" id="audioUpload" accept=".mp3,.wav" class="hidden">
48
+ <div class="flex items-center justify-center bg-gray-700 hover:bg-gray-600 rounded-lg p-4 transition-all duration-300">
49
+ <i data-feather="upload" class="mr-2"></i>
50
+ <span>Choose Audio File</span>
51
+ </div>
52
+ </label>
53
+ <button id="playBtn" class="bg-purple-600 hover:bg-purple-700 px-6 py-3 rounded-lg font-medium flex items-center justify-center transition-colors">
54
+ <i data-feather="play" class="mr-2"></i>
55
+ Play
56
+ </button>
57
+ </div>
58
+
59
+ <div class="mb-6">
60
+ <div class="flex justify-between text-sm text-gray-400 mb-1">
61
+ <span id="currentTime">0:00</span>
62
+ <span id="totalTime">0:00</span>
63
+ </div>
64
+ <input type="range" id="progressBar" class="w-full h-2 bg-gray-700 rounded-lg appearance-none cursor-pointer" value="0">
65
+ </div>
66
+
67
+ <div class="grid grid-cols-2 sm:grid-cols-3 gap-3">
68
+ <button class="synesthesia-mode bg-gradient-to-br from-blue-500 to-indigo-600 px-4 py-3 rounded-lg font-medium transition-all" data-mode="chromesthesia">
69
+ Chromesthesia
70
+ </button>
71
+ <button class="synesthesia-mode bg-gradient-to-br from-green-500 to-teal-600 px-4 py-3 rounded-lg font-medium transition-all" data-mode="tactile">
72
+ Auditory-Tactile
73
+ </button>
74
+ <button class="synesthesia-mode bg-gradient-to-br from-red-500 to-orange-600 px-4 py-3 rounded-lg font-medium transition-all" data-mode="spatial">
75
+ Spatial Harmony
76
+ </button>
77
+ </div>
78
+ </div>
79
+
80
+ <div id="visualizerContainer" class="relative w-full max-w-4xl aspect-square rounded-xl overflow-hidden shadow-2xl">
81
+ <div id="audioCanvas"></div>
82
+ <div class="absolute inset-0 flex items-center justify-center bg-black bg-opacity-30" id="uploadPrompt">
83
+ <div class="text-center p-6">
84
+ <i data-feather="music" class="w-12 h-12 mx-auto mb-4 text-gray-300"></i>
85
+ <h3 class="text-xl font-medium mb-2">Upload an audio file</h3>
86
+ <p class="text-gray-300">Supported formats: MP3, WAV</p>
87
+ </div>
88
+ </div>
89
+ </div>
90
+ </main>
91
+
92
+ <footer class="py-6 px-4 text-center text-gray-400 text-sm">
93
+ <p>Based on neurological research into synesthesia and cross-modal perception</p>
94
+ </footer>
95
+
96
+ <script>
97
+ feather.replace();
98
+
99
+ // Initialize Vanta.js background
100
+ VANTA.GLOBE({
101
+ el: "#vanta-bg",
102
+ mouseControls: true,
103
+ touchControls: true,
104
+ gyroControls: false,
105
+ minHeight: 200.00,
106
+ minWidth: 200.00,
107
+ scale: 1.00,
108
+ scaleMobile: 1.00,
109
+ color: 0x8a4fff,
110
+ backgroundColor: 0x09090b,
111
+ size: 0.8
112
+ });
113
+
114
+ // Audio visualization setup
115
+ let audioContext, analyser, dataArray, source;
116
+ let isPlaying = false;
117
+ let currentMode = 'chromesthesia';
118
+ let audioElement, gainNode;
119
+
120
+ document.getElementById('audioUpload').addEventListener('change', function(e) {
121
+ const file = e.target.files[0];
122
+ if (file) {
123
+ document.getElementById('uploadPrompt').style.display = 'none';
124
+
125
+ if (audioElement) {
126
+ audioElement.pause();
127
+ audioElement = null;
128
+ }
129
+
130
+ audioElement = new Audio(URL.createObjectURL(file));
131
+ document.getElementById('playBtn').innerHTML = '<i data-feather="play" class="mr-2"></i>Play';
132
+ feather.replace();
133
+
134
+ // Set up audio context on first interaction
135
+ if (!audioContext) {
136
+ audioContext = new (window.AudioContext || window.webkitAudioContext)();
137
+ analyser = audioContext.createAnalyser();
138
+ analyser.fftSize = 256;
139
+ gainNode = audioContext.createGain();
140
+
141
+ source = audioContext.createMediaElementSource(audioElement);
142
+ source.connect(gainNode);
143
+ gainNode.connect(analyser);
144
+ analyser.connect(audioContext.destination);
145
+
146
+ dataArray = new Uint8Array(analyser.frequencyBinCount);
147
+ }
148
+
149
+ // Set up duration display
150
+ audioElement.addEventListener('loadedmetadata', function() {
151
+ const duration = audioElement.duration;
152
+ const minutes = Math.floor(duration / 60);
153
+ const seconds = Math.floor(duration % 60).toString().padStart(2, '0');
154
+ document.getElementById('totalTime').textContent = `${minutes}:${seconds}`;
155
+ });
156
+
157
+ // Update progress bar
158
+ audioElement.addEventListener('timeupdate', function() {
159
+ const currentTime = audioElement.currentTime;
160
+ const duration = audioElement.duration || 1;
161
+ const minutes = Math.floor(currentTime / 60);
162
+ const seconds = Math.floor(currentTime % 60).toString().padStart(2, '0');
163
+ document.getElementById('currentTime').textContent = `${minutes}:${seconds}`;
164
+ document.getElementById('progressBar').value = (currentTime / duration) * 100;
165
+ });
166
+
167
+ // Set up progress bar scrubbing
168
+ document.getElementById('progressBar').addEventListener('input', function() {
169
+ if (audioElement.duration) {
170
+ audioElement.currentTime = (this.value / 100) * audioElement.duration;
171
+ }
172
+ });
173
+
174
+ // Initialize p5.js visualizer
175
+ new p5(function(p) {
176
+ let particles = [];
177
+
178
+ p.setup = function() {
179
+ const container = document.getElementById('visualizerContainer');
180
+ const canvas = p.createCanvas(container.clientWidth, container.clientHeight);
181
+ canvas.parent('audioCanvas');
182
+
183
+ // Initialize particles based on mode
184
+ initParticles();
185
+ };
186
+
187
+ p.windowResized = function() {
188
+ const container = document.getElementById('visualizerContainer');
189
+ p.resizeCanvas(container.clientWidth, container.clientHeight);
190
+ };
191
+
192
+ function initParticles() {
193
+ particles = [];
194
+ const count = currentMode === 'spatial' ? 500 : 200;
195
+
196
+ for (let i = 0; i < count; i++) {
197
+ particles.push({
198
+ x: p.random(p.width),
199
+ y: p.random(p.height),
200
+ size: p.random(2, 6),
201
+ speed: p.random(0.5, 2),
202
+ angle: p.random(p.TWO_PI),
203
+ color: getRandomColor(),
204
+ freqIndex: Math.floor(p.random(analyser.frequencyBinCount))
205
+ });
206
+ }
207
+ }
208
+
209
+ function getRandomColor() {
210
+ if (currentMode === 'chromesthesia') {
211
+ return p.color(
212
+ p.random(100, 255),
213
+ p.random(100, 255),
214
+ p.random(100, 255),
215
+ p.random(150, 220)
216
+ );
217
+ } else if (currentMode === 'tactile') {
218
+ return p.color(
219
+ 100 + p.random(100),
220
+ 100 + p.random(100),
221
+ 255,
222
+ p.random(180, 220)
223
+ );
224
+ } else {
225
+ return p.color(
226
+ 255,
227
+ 100 + p.random(100),
228
+ 100 + p.random(100),
229
+ p.random(180, 220)
230
+ );
231
+ }
232
+ }
233
+
234
+ p.draw = function() {
235
+ p.background(0, 0, 0, 25);
236
+
237
+ if (analyser && audioElement && !audioElement.paused) {
238
+ analyser.getByteFrequencyData(dataArray);
239
+
240
+ // Draw particles based on current mode
241
+ if (currentMode === 'chromesthesia') {
242
+ drawChromesthesia();
243
+ } else if (currentMode === 'tactile') {
244
+ drawTactile();
245
+ } else {
246
+ drawSpatial();
247
+ }
248
+ }
249
+ };
250
+
251
+ function drawChromesthesia() {
252
+ p.noStroke();
253
+ const centerX = p.width / 2;
254
+ const centerY = p.height / 2;
255
+
256
+ particles.forEach(part => {
257
+ const freqValue = dataArray[part.freqIndex] / 255;
258
+
259
+ // Move particle in circular pattern
260
+ part.angle += part.speed * 0.01;
261
+ part.x = centerX + Math.cos(part.angle) * (100 + freqValue * 200);
262
+ part.y = centerY + Math.sin(part.angle) * (100 + freqValue * 200);
263
+
264
+ // Size and color change with frequency
265
+ const pulseSize = part.size * (1 + freqValue * 2);
266
+ part.color.setAlpha(150 + freqValue * 105);
267
+
268
+ p.fill(part.color);
269
+ p.circle(part.x, part.y, pulseSize);
270
+ });
271
+ }
272
+
273
+ function drawTactile() {
274
+ p.noStroke();
275
+ const centerY = p.height / 2;
276
+
277
+ particles.forEach(part => {
278
+ const freqValue = dataArray[part.freqIndex] / 255;
279
+
280
+ // Vertical wave motion
281
+ part.y = centerY + Math.sin(p.frameCount * 0.02 * part.speed + part.x * 0.01) * 100 * freqValue;
282
+
283
+ // Pulsing effect
284
+ const pulseSize = part.size * (1 + freqValue * 3);
285
+ part.color.setAlpha(180 + freqValue * 75);
286
+
287
+ p.fill(part.color);
288
+ p.ellipse(part.x, part.y, pulseSize, pulseSize * 0.6);
289
+
290
+ // Move horizontally slightly
291
+ part.x += part.speed * 0.1;
292
+ if (part.x > p.width) part.x = 0;
293
+ });
294
+ }
295
+
296
+ function drawSpatial() {
297
+ p.noStroke();
298
+ const centerX = p.width / 2;
299
+ const centerY = p.height / 2;
300
+
301
+ particles.forEach(part => {
302
+ const freqValue = dataArray[part.freqIndex] / 255;
303
+
304
+ // 3D effect with perspective
305
+ const z = freqValue * 5;
306
+ const scale = p.map(z, 0, 5, 0.2, 1.5);
307
+ const x = centerX + (part.x - centerX) * scale;
308
+ const y = centerY + (part.y - centerY) * scale;
309
+
310
+ part.color.setAlpha(100 + freqValue * 155);
311
+ p.fill(part.color);
312
+ p.circle(x, y, part.size * scale);
313
+
314
+ // Connect particles that are close
315
+ p.stroke(part.color);
316
+ p.strokeWeight(0.5);
317
+ particles.forEach(other => {
318
+ if (other !== part && p.dist(x, y, other.x, other.y) < 50 * scale) {
319
+ p.line(x, y, other.x, other.y);
320
+ }
321
+ });
322
+ });
323
+ }
324
+ }, 'audioCanvas');
325
+ }
326
+ });
327
+
328
+ // Play/pause button
329
+ document.getElementById('playBtn').addEventListener('click', function() {
330
+ if (audioElement) {
331
+ if (audioElement.paused) {
332
+ audioElement.play();
333
+ this.innerHTML = '<i data-feather="pause" class="mr-2"></i>Pause';
334
+ if (audioContext.state === 'suspended') {
335
+ audioContext.resume();
336
+ }
337
+ } else {
338
+ audioElement.pause();
339
+ this.innerHTML = '<i data-feather="play" class="mr-2"></i>Play';
340
+ }
341
+ feather.replace();
342
+ }
343
+ });
344
+
345
+ // Synesthesia mode buttons
346
+ document.querySelectorAll('.synesthesia-mode').forEach(button => {
347
+ button.addEventListener('click', function() {
348
+ currentMode = this.dataset.mode;
349
+ document.querySelectorAll('.synesthesia-mode').forEach(btn => {
350
+ btn.classList.remove('ring-2', 'ring-white');
351
+ });
352
+ this.classList.add('ring-2', 'ring-white');
353
+
354
+ // Reinitialize visualization (handled in p5 instance)
355
+ });
356
+ });
357
+ </script>
358
+ </body>
359
  </html>
prompts.txt ADDED
@@ -0,0 +1,49 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Create a web-based audio visualizer that generates immersive, procedurally created visuals based on uploaded audio files. The system should leverage medically backed data on synesthesia, auditory perception, and cross-modal sensory integration to simulate how people with synesthesia may “see” music.
2
+
3
+ Core Requirements:
4
+
5
+ Audio Input & Processing
6
+
7
+ Users can upload .wav or .mp3 files.
8
+
9
+ The engine extracts frequency bands, amplitude, tempo, and timbral qualities.
10
+
11
+ Use FFT or similar techniques to generate real-time analysis.
12
+
13
+ Medical/Scientific Backing
14
+
15
+ Base color, motion, and particle mapping rules on published neurological and psychological studies of synesthesia (e.g., sound-color, pitch-shape, rhythm-motion correlations).
16
+
17
+ Allow configurable “synesthesia modes” such as Chromesthesia (sound → color), Auditory-Tactile (sound → vibration patterns), etc.
18
+
19
+ Visualization Engine
20
+
21
+ Render visuals in WebGL (Three.js or similar) for smooth, GPU-accelerated performance.
22
+
23
+ Procedurally generate glowing particle fields, fluid motions, and geometric transformations tied to music intensity and frequency.
24
+
25
+ Ensure visuals evolve dynamically over time, not just repeating loops.
26
+
27
+ UI/UX
28
+
29
+ Simple upload interface for audio files.
30
+
31
+ Play/Pause controls with timeline scrubbing.
32
+
33
+ Option to toggle different visualization “modes” (scientifically inspired presets).
34
+
35
+ Responsive design so it works on desktop and mobile browsers.
36
+
37
+ Extendability
38
+
39
+ Architecture should allow adding new medically informed mappings later (e.g., linking timbre → texture, harmony → spatial depth).
40
+
41
+ Export visuals as high-resolution video or shareable web links.
42
+
43
+ Deliverables:
44
+
45
+ A web application prototype (React + Three.js or similar) hosted in the browser.
46
+
47
+ A modular procedural visualization engine that maps sound features to medically grounded sensory equivalents.
48
+
49
+ Clean, well-documented code so mappings can be extended with new research data.