Currently, we recognize a variable LOCAL_DIR (that can be set in cdap-env.sh) and pass that to the JVM with -Duser.dir=$LOCAL_DIR. That only affects files that are created by CDAP in its working directory. However, CDAP will still write files to /tmp, which can be a problem is the temp file system is small.
Suggestion is to also recognize a TEMP_DIR and pass that in with -Djava.io.tmpdir=$TEMP_DIR.
The current work-around is to add:
OPTS="$OPTS -Djava.io.tmpdir=/my/tmp/dir" in cdap-env.sh. However, the OPTS variable is also set by common-env.sh and users might accidentally overwrite that, instead of appending to it. Also, it is nicer not have users configure JVM options directly.