AltME groups: search

I don't really need anything but having the ability to parse the 
first 100 lines of a file and doing that many times, so I don't care 
so much about continuation. This is for real-time previews of large 
CSV files (> 10000 lines).

The main problem with /part is that the current code reads the whole 
file into memory before parsing, and the parsing itself has miniscule 
overhead compared to the file overhead. Really doing /part properly 
might require incremental file reading, to the extent that that works 
(how well does it work for the http scheme on R3?).

That's fine by me, as I read the file into memory once due to the 
need for one-time UTF-8 conversion, so that will happen outside LOAD-CSV.

LOAD has the same problem on R2 and R3. The continuation returned 
would be an offset reference to the entire data in the file, at the 
position after the part parsed.

http://www.rebol.org/view-script.r?script=csv-tools.rupdated, with 
the new LOAD-CSV /part option.

The LOAD-CSV /part option takes two parameters:
- count: The maximum number of decoded lines you want returned.

- after: A word that will be set to the position of the data after 
the decoded portion, or none.


If you are loading from a file or url then the entire data is read, 
and after is set to a position in the read data. If you are converting 
from binary then in R2 after is set an offset of an as-string alias 
of the binary, and in R3 after is set to an offset of the original 
binary. R3 does binary conversion on a per-value basis to avoid having 
to allocate a huge chunk of memory for a temporary, and R2 just does 
string aliasing for the same reason. Be careful to expect that if 
you are passing the value assigned to after to anything else than 
LOAD-CSV (which can handle it either way).

I considered making a /strict option to make it trigger errors in 
that case, but then reread the RFC and checked the behavior again, 
and realized that noone took the spec that strictly. Most tools either 
behave exactly the same as my LOAD-CSV (because that's how Excel 
behaves), or completely fail when there are any quotes in the file, 
like PARSE data "," and PARSE/all data ",".

Have you looked into the native type formatting of bcp? It might 
be easier to make a more precise data file that way.

It uses a format file, it is very strict, but no chance to set a 
quote char for fields.

trying to get info on a file via ftp using to long version of teh 
port spec as my user name is an email address:

fport: [
    scheme: 'FTP
    host: "ftp.example.com"
    target: %/file.txt
    user: "bill@ example.com"
    pass: "vbs"
]

I can read it with "read fport"

but I can not get other info from it like:

print modified? fport

Whats the method here?

Again, this might be a Graham question: I'm still working with that 
video encoder which uses http to communicate. They have a .cgi script 
which downloads the recorded video file from the internal SD card 
to the requester. My problem is the content I receive is somehow 
different than the files which I can download via a browser and of 
course will not play. I still using your http-tools to GET/POST. 
My initial thought was that  data returned is somehow being translated. 
Any thoughts?

I am making a simple (I hope) worfkflow prototype and want to use 
REBOL objects which I can SAVE and LOAD.  A workflow object!  to 
have a node-map collection in it of simple nodes of the workflow 
graph.   Source ->A -> B -> SINK where the workflow knows about the 
next node and status. Externally there is UI to support the work 
part ... which is URL data on a given node.   Looks like it fits 
into Cheyenne RSP well - maybe zmq when I get a bit further along. 
  Save a flow in process as a .txt file using SAVE/ALL filename.txt 
work-flow-instance.   But no success with work-flow-instance: LOAD 
filename.txt        Do I have to BIND on LOAD to re-instantiate the 
object?

The SAVE is a workflow function persist: does [save/all to-file rejoin 
[%data/ id ".txt"] self]

the change history of what? r3 file storage?

I'm having problems renaming a file on a USB stick with R2 under 
Windows. It reports "cannot access". R3 does not have this problem. 
Can anyone reproduce this?

ok, that is interesting. possibly file system related?

the mezz source says that if the file is not found, then it can't 
be accessed.

Henrik, I have no problem with renaming a file on USB stick...tested 
under WindowsXP SP3

Henrik, I can read the file with REBOL command READ in R2.

it works, if I open the port and find the file by hand

Reading the source for RENAME, it rename the file by changing its 
name in the directory file. And this doesn't work for some reason 
with USB sticks under OS X using R2.

the CHANGE works, if I do it by hand. Then I can rename the file.

rename capabilities in file handling do not normally allow paths 
to be used (in the OS itself).  otherwise these are a called 'move 
file operations. 


e.g. if you try using paths with rename in the DOS shell, you get 
errors.

In answer to your comments link above:
- Syntax errors are triggered before semantic errors: 1.3, 11

- Words that start with + and - are special because of potential 
ambiguity with numbers: 1.1

- Arrows are only allowed in the special-case arrow words, not generally: 
1.2, 1.3, 4

- %: is ambiguous - it could be a file that wouldn't work on any 
OS, or the set-word form of %, so an error splits the difference: 
10.2
- Fixed already: 2.2 for arrows in R3, 7, 13


Some of the rest are related to http://issue.cc/r3/537and others 
have been reported already. If you want 10.2 to not trigger an error, 
it is more likely to be accepted as a set-word than a file. Thanks 
for these, particularly the lit-word bugs.

A bug?

>> cur: what-dir

== %/Users/peter/

>> cd %Code/Rebol

== %/Users/peter/Code/Rebol/
>> cd cur

** Access Error: Cannot open /Users/peter/Code/Rebol/cur/

** Near: change-dir to-file path

>> change-dir cur

== %/Users/peter/
Though this works:

>> cd :cur
== %/Users/peter/

Geomol: i was just using new-line recently for example to create 
a simple "json database". i was saving json-to-rebol object hierarchies 
into plain text files, but before that i converted each object into 
pure blocks, so it has less brackets... no #[object! ...] shit. but 
it destroyed the new-lines, so i had to but them back manually, so 
the text file is still human processable.

This is about the HTTP scheme, but I can't find a group for R2 schemes.


Does anyone have a patch for the HTTP scheme that handles 204 (No 
Content) responses where no headers are returned? The standard scheme 
throws an error as there are no headers to parse. Here is the 'success 
case handler:

        success: [
            headers: make string! 500

            while [(line: pick port/sub-port 1) <> ""] [append headers join line 
            "^/"]

            port/locals/headers: headers: Parse-Header HTTP-Header headers
            port/size: 0

            if querying [if headers/Content-Length [port/size: load headers/Content-Length]]

            if error? try [port/date: parse-header-date headers/Last-Modified] 
            [port/date: none]

            port/status: 'file
    ]


For anyone familiar with the scheme, would the proper behavior be 
to set all related 'port fields to zero or none? e.g.

            port/locals/headers: headers: none
            port/size: 0
            port/date: none
            port/status: none

It's kinda interesting, how complexity sneak into a language like 
REBOL. I have never used constructs. I save REBOL data and code to 
disk all the time. I even created a file system/database, that is 
all about saving and loading data and code. I get along using a combination 
of simple REBOL functions like SAVE, LOAD, REDUCE and DO.

As suggested by some people, I am making the COMMENT directive standard, 
improving all the directives, and enhancing the way how INCLUDE generates/uses 
errors. When INCLUDE is traversing a large set of files, I feel it 
convenient not only to get an error, but also the file, where the 
error occurred.

That is possible by either


- enhancing the error to contain the information about the file, 
where it occurred

- storing the name of the culprit file somewhere else, not into the 
error itself

The former situation (the information about the "culprit file" is 
stored in the error) looks as follows, currently:

performing localization
** User Error: INCLUDE
** Near: do last-error: make error! spec


The trouble is, that the present error-forming code does not show 
all the attributes. If you examine the error on your own, you get:

print mold disarm last-error

make object! [
    code: 802
    type: 'user
    id: 'message
    arg1: 'syntax
    arg2: %gui/include.r
    arg3: [
        id: missing
        arg1: "end-of-block"
        arg2: "["
        arg3: none
        near: "(line 949) ]"
    ]
    near: [do last-error: make error! spec]
    where: none
]


, which shows all the data as "stored" in the error, which is referred 
(for convenience) by the LAST-ERROR variable

aha, correction, the current look of the error is as follows:

>>print mold disarm last-error

make object! [
    code: 802
    type: 'user
    id: 'message
    arg1: "INCLUDE"
    arg2: 'syntax
    arg3: [
        file: %actions/tabs/data.r
        id: missing
        arg1: "end-of-block"
        arg2: "["
        arg3: none
        near: "(line 949) ]"
    ]
    near: [do last-error: make error! spec]
    where: none
]

The second option would be to not "enhance" the error, in which case 
it might look like:

** Syntax Error: Missing [ at end-of-script
** Near: (line 949) [

, and examining the error we would get:

make object! [
 id: missing
        arg1: "end-of-block"
        arg2: "["
        arg3: none
        near: "(line 949) ]"
]


here, clearly, the information that it was an error in the %actions/tabs/data.r 
file is missing, but the "standard" error message is more informative. 
The missing CULPRIT-FILE information could be supplied by defining 
a CULPRIT-FILE variable for that purpose. Any preference(s) which 
alternative you might prefer?

Summary of the advantages of the first approach:

+ the file information is present in the error itself

Disadvantages:


- the error is "too complicated" for the interpreter to display the 
important informations

- the error has to be "intercepted" by TRY for the INCLUDE to be 
able to "enhance" it.

- also, since the INCLUDE works recursively, the TRY is used many 
times, and the code needs to take care, that the "enhancement" occurs 
only once

The second approach has got the following advantages:


+ no need to "intercept" the error, since no "error enhancement" 
needs to be done

+ the error is displayed by the interpreter in a standard way, the 
user needs just to get the CULPRIT-FILE name elsewhere

Disadvantages:


- the error does not contain the CULPRIT-FILE information, which 
is important, thus, the user needs to look for it elsewhere

As an example, check this:


["You do not have sufficient rights to delete the '%1" file." "database.r"]


This is easy to translate, since only the substitution strings needs 
a translation, while the argument does not. If you know how to translate 
the substitution string, the argument string does not matter at all.

So, the TRANSLATE function just replaces the substitution string 
by a different language version, leaving the argument as-is, e.g.:


["Sie verfuegen nicht die ausreichende Zugriffsrechte fuer das File 
'%1' zu loeschen." "database.r"]

(forgive my attempt, I bet it is not a correct German)

Originally, the block might have been generated by something like:


my-substitution: reduce ["You do not have sufficient rights to delete 
the '%1" file." to string! file]

type? load form [1] ; == integer! (i.e. not block!)


Yeah, that's a pity, I think. I would prefer LOAD to always return 
a block, so the result from LOAD could always be sent to e.g. PARSE. 
I guess, it's made this way to kinda let LOAD and SAVE reflect each 
other. But that doesn't quite make sense, as we can't save to a string. 
And LOAD can load an empty file giving an empty block, while we can't 
save an empty file with SAVE, afaik.

I want to share with you an "interoperability problem" I encountered. 
In Windows (at least in not too old versions) there are two versions 
of string-handling functions:

- ANSI (in fact using a codepage for latin charset)
- widechar (in fact UNICODE, restricted to 16 bits, I think)


It looks, that Apple OS X "prefers" to use decomposed UNICODE, also 
known as UTF-8MAC, I guess. That means, that it e.g. for a Robert's 
file it generates a filename looking (in a transcription) as follows:

%"Mu^(combining-umlaut)nch.r"

As far as the UNICODE goes, this is canonically equivalent to

%"M^(u-with-umlaut)nch.r"

, but:


- Windows don't consider these file names equivalent, i.e. you can 
have both in one directory

- When using the former, the ANSI versions of Windows system functions 
"translate" the name to: %"Mu^(umlaut)nch.r"

-- the %"Mu^(umlaut)nch.r" is a third file name, distinct from both 
of the above, so, if the R2 reads it in a directory, it is unable 
to open it

Linux also would not consider them equivalent. I understand why Mac 
OS is always canonicizing file names, but, they chose the most stupid 
way to do it, and it's a pain in the ass most of the time. In the 
end, I prefer Linux where you can end up with two files named in 
a way that looks exactly the same, but that has a file system that 
behaves in a predictable way.

why does this not work:


unless any [exists? file1 exists? file2 exists? file3][print "missing 
file"]

but this does: 



 if any [not exists? file1 not exists? file2 not exists? file3][print 
 "missing file"]

if I have a file with url-encoded chars in it, what's the fastest 
way to decode them?

Ran code like the following against a 600MB file of 500,000+ lines:

	file: read/lines %log.csv
	foreach line file [
		remove/part line 1000
	]


and was surprised that the foreach loop was near instantaneous (I'd 
assumed 500,000+ removes would at least take a second or two). I'm 
not complaining, just curious. ;)

It certainly worked (file size was 400MB not 600MB).

I do something similar, generating 10'000 lines and writing to a 
file takes 5-6 seconds to complete on my Intel Core 2 Duo E7400 (2.8 
Ghz) 3GB RAM.

500'000 lines a bit too much but you read the whole file into memory 
so it is possible. Your PC is a good one as well I think.

which means 10 times better than mine :) its normal to have that 
performance with 8GB RAM + SSD I think. file & block operations & 
foreach is fast on R2 in my experience.

I need to extract the data from an image file so it does not include 
the "64#" header and I just have the 64 bit encoding: 

64#{
/9j/4faARXhpZgAATU0AKgAAAAgABwEPAAIAAAAESFRDAAEQAAIAAAAIAAAAYgEa
AAUAAAABAAAAagEbAAUAAAABAAAAcgEoAAMAAAABAAIAAAITAAMAAAABAAEAAIdp
is3eIoxUdG7n/9k=
}


I just wnat the stuff between the quotes but as its a binary I can't 
seem to parse it or extract it with other methods like a text file.

I have uploaded my latest EXIF-parser version at github - https://github.com/Oldes/rs/blob/master/projects/exif-parser/latest/exif-parser.r

To sort files you can use for example:

dir: %/e/DCIM/100CANON/
t: now/time/precise
result: copy []
foreach file read dir [
	error? try [
		ctx-exif/parse-file dir/:file
		exifdate: ctx-exif/get-tag-value 306
		repend result [exifdate dir/:file]
	]
]

sort/skip result 2

print ["sorted" (length? result) / 2 "files in" now/time/precise 
- t]
result

;>>sorted 120 files in 0:00:00.153

Seems REBOL/View is required:

>> do/args %rsc.r "%tests/hello.reds"
** Script Error: Feature not available in this REBOL
** Where: context
** Near: file-header: make struct! [

We won't want to be limited to one-file programs forever :)

amacleod: It's mentioned in the README file:

4) The resulting binary is in red-system/builds/hello.exe, open a 
DOS console and run it.

Here's the direct download link: ttp://software.intel.com/file/7242 
(~27MB)

sorry: http://software.intel.com/file/7242

ELF seems to be a bit smarter about page loading, it doesn't require 
explicit padding to page boundaries in the file itself, so can generate 
smaller exe. (It's possible with PE too with some hacking).

[[kaj-:-syllable]:~/Red]readelf -l empty

                                                                                                                                                                                                        
Elf file type is EXEC (Executable file)
Entry point 0x8048074
There are 2 program headers, starting at offset 52

                                                                                                                                                                                                        
Program Headers:

  Type           Offset   VirtAddr   PhysAddr   FileSiz MemSiz  Flg 
  Align

  LOAD           0x000000 0x08048000 0x08048000 0x00184 0x00184 R E 
  0x1000

  LOAD           0x000184 0x08048184 0x08048184 0x00032 0x00032 RW 
   0x1000
[[kaj-:-syllable]:~/Red]

builds/hello:     file format elf32-i386

Sections:
Idx Name          Size      VMA       LMA       File off  Algn
  0 .shstrtab ...

The code uses the new preprocessor. It's starting to look quite nice, 
and I've split off the examples into separate files, to the binding 
is now a proper separate library/#include file

none: I think Kaj has already defined it as null in his 0MQ wrapper. 
:-) So, I could add it in the common runtime file, it costs nothing 
(which is the least for none). :-)

I've listed 4 peoples in the README file, but Brahim's and Arnaud's 
source code never made it in the repository (project stopped before), 
Volker's contribution is a 10-lines C function. That's hardly a "4 
people working for it". I have written 99,99% of the released source 
code.

so red is compiled but then it's systeme dependant and we can't test 
small chunks of code like in R2 consol in my opinion one of the strong 
point of rebol was this ability to open it's consol test an epurated 
bunch of code and then once working enhance it on our script file. 
I would like red somehow to get that  ability maybe it will be possible 
in the IDE or as a side stuff. For me the 2 best points of rebol 
were reflexivity code <--> data code = data data = code and parse. 
Even if I didn't fully understand parse I made a great use in my 
productions in rebol script VID oriented of the reflexivity code 
<---> data. All the other arguments of rebol are not really interresting 
since they are double sided and so not objective and so just a matter 
of mood and point of view.

One very strong feature in REBOL is that functions can take an argument 
of more than one datatype. You can e.g. READ a file, a url or a port. 
Do you see a solution for this in Red?

I forgot that such feature existed in AltME (as for Calendar and 
File Sharing).

BTW, I was thinking these last days that I maybe should allocate 
a week before starting the work on Red's compiler, to implement an 
experimental JVM bytecode emitter (and a .class file format). The 
reason for that is to have a 2nd emitter (in addition to IA32) that 
is addressing a very different architecture. This should help make 
a cleaner abstraction layer for upcoming emitters (especially those 
not written by me).

I might alternatively pick up CLR for that task, thought (less work 
for the file format as it is basically PE).

Could you try changing the base address in ELF.r? It is a the top 
of the file:
context [
	defs: [
		image [
			base-address	134512640	; #{08048000}
		]

Awi, that would be great, I would love to see Red run on such platform 
but I'm afraid Red's memory footprint would be a bit too high for 
a 60KB of RAM (but this might be specific to the netduino platform?). 
OTOH, Red/System compiler should work nicely there. If you want to 
help for such port, we need two things:


- a target file format emitter (I guess it would be the firmware 
file format)
- a native ARM7 code emitter as Red/System back-end. 


Let me know if you want to start working on these parts, I would 
provide additional documentations.

but it results in the same problem of generating wrong file offsets

(because other code parts re-use data-ptr to calculate the file offset)

I think Steve Shireman has done TCP stacks in the past. 


My only recent thought, related to OS dependencies and such, was 
what it would be like if your only interface to files was a memory 
mapped file interface. Thinking about languages and desktop/server 
systems, not the embedded stuff.

For some functionality, I need to have access to the stdin, stdout 
and stderr identifiers. For syscalls, they're simple integers, but 
in the C library, they're pointers to file descriptors. There's currently 
no way to get their values

Implemented remaining string parsing and file functions

#include was changed to only include a file once a little while ago. 
There was a comment to that effect in the commit notes.

Steeve: which system image file have you used?

You just do from menu: File->Open.

I was afraid the gets type functions are implemented as macros, so 
that may well be the problem. The thing is that I can't fix it by 
using the fgets type functions, because Red can't import the standard 
file descriptors such as stdin yet

Kaj: on Windows, it is possible to use _get_osfhandle() to retrieve 
the file handle for stdin/out/err (http://msdn.microsoft.com/en-us/library/ks2530z6.aspx)

Is it possible to specify a build target per file? If I don't want 
to use %builds/ folder, but for example %builds/project/ and specify 
it in the project.reds file header.

*** Compilation Error: multiple type casting not allowed
*** in file: %runtime/../library/C-library.reds
*** in function: temporary-name
*** at:  [as c-string! null]

So flushing the C I/O streams is the solution:
  
    print-1 "int %d" 1
    flush-file null
    print "hello"

works fine.

Compiling /users/administrator/Red/Red-ZeroMQ-binding/examples/reply-server.reds 
...
*** Compilation Error: wrong return type in function: receive 
*** expected: [message!], found: [none] 

*** in file: %/users/administrator/Red/Red-ZeroMQ-binding/examples/reply-server.reds 
*** in function: receive
*** at:  []

*** Compilation Error: undefined symbol: message! 

*** in file: %/users/administrator/Red/Red-ZeroMQ-binding/examples/reply-server.reds 
*** at:  [message! 
    print-newline

message! is aliased in ZeroMQ-binding.reds, but it doesn't seem to 
carry over to the source file where it's included

# builds/hello 
ELF binary type "0" not known.
builds/hello: Exec format error. Binary file not executable.

But I can upload a file to FTP now

No, I can upload one file from disk exclusively. The binding is two 
days old

I've implemented a callback framework and used that to implement 
sending a data block from memory, instead of direct from a file

I've extended the read function in the cURL binding to store a file 
being received directly to disk, instead of just printing it

GDB on Syllable currently only supports real-time debugging, as crashed 
applications do not generate a 
core" file (crash dump)."

It took me a few hours to write the file emitter but a full day of 
debugging to make it work. It seems that the issue was related to 
stricter stack alignment requirement on OS X (16 bytes alignment) 
and a different ABI for syscalls. 


It currently works thanks to a few hacks, but I need to extend the 
IA-32 emitter to implement these new requirements in order to make 
it output correct code for OS X.

Red/System relies on shared libraries dynamic linking at load time. 
CPU and ABI specific calling conventions are handled by respectively 
Red's target emitters and file emitters. I don't see how libffi could 
make it simpler or better. From what I understand from FFI (very 
short) descriptions, it is more suitable for high-level languages 
that can't or don't want to deal with low-level interfaces.

Cross-posting jocko's reply from RebelBB french forum: 

There are two steps in this DLL: 


- first send a request to the Google api, for instance : http://translate.google.com/translate_tts?tl=en&q=hello+world
Google returns an audio file : translate_tts.mp3


- second, automatically open this file and play it. I use a DirectXShow 
filter (windows only), directly able to render the url. In the case 
of Linux, cUrl could do the first step, and, for instance VLC the 
second

Not that any of you would have issues with creating such a thing 
but here is an R2 version of the Colineau's (Jocko) Google translate 
app he created in Red. (Note, I didn't add all of the routines but 
if you take a look at Colineau's code it's all there.)  Also, the 
female voice (use gTTS function instead of TTS function) is much 
better than the male in my opinion unless you want to hear "This 
is Amiga Speaking." and feel nostalgic.

rebol [
	title: {googletts.r}
	date: 20-aug-2011
	usage: {gtts "Hello World." or tts "Hello World."}	
]

lib: load/library %tts-jc.dll

TTS: make routine! [
	lpStr [string!]
	return: [integer!]		
] lib "TTS"

gTTS: make routine! [
	lpStr [string!]
	return: [integer!]		
] lib "gTTS"

-----

I created some nice memory tools for my son who is in law school 
with this by setting up the string and tweaking it and then recording 
it (I use Sound Forge). If I get some free time I'd like to create 
a dialect so that I can make an interactive tool with visual reinforcements. 
As I mentioned, you have to tweak the words and punctuation and that 
creates a problem with just reading the text normally, hence I'll 
require a mechanism to sort all that out.

Oh, the dll is in the http://www.colineau.fr/rebol/downloads/demoTTS_Red.zip
file

I mean, instead of having to call it from every user script, it could 
be called as last expression in %GTK.reds source file directly?

It demoes mouse events, drawing, program arguments and loading a 
bitmap file

I've added a sample.bmp file from SDL for easy access

Runtime Error 1: access violation
in file:

that's all.  No filename

*** Compilation Error: a variable is already using the same name: 
window
*** in file: %tests/hello-GTK-world.reds

Kaj, I downloaded all files again and tried it on another PC with 
Win 7 (x64). Same compilation error, but I also noticed while running 
do/args %rsc.r "%tests/hello-GTK-world.reds"

The first run


*** Warning: type casting from pointer! to pointer! is not necessary
*** in: %tests/hello-GTK-world.reds
*** at:  [as integer! :gtk-quit null

*** Compilation Error: invalid struct member alias in: type
*** in file: %tests/hello-GTK-world.reds
*** in function: window
*** at:  [type system/alias/gdk-image! [

The next runs


*** Compilation Error: a variable is already using the same name: 
window
*** in file: %tests/hello-GTK-world.reds
*** in function: window
*** at:  [

graham, topaz is a silicate based gemstone, with the correct impurities 
it could morphe to topaz_red.

as you can see here: http://en.wikipedia.org/wiki/File:Topaz_red.jpg 
  :-)))

myvar=`ls  /home/myname/mydir`  foreach file myvar [ print "file 
name: " + file ]

onetom: I used Keynote, it can export to a few formats, including 
HTML (by just making PNGs, so it gets pretty big). I can also provide 
the keynote file if useful. (Unfortunately, I was very tight on time 
and was unable to do this in Topaz itself.)